Display device and control method

Abstract

To provide a display device, etc. which, when a plurality of screens are displayed so as to be superimposed on each other, can suitably perform processing for an operation.SOLUTION: A display device includes a display part and a control part. The control part displays on the display part a first display screen which can include a transmissive region and a second display screen which is displayed so as to be positioned behind the first display screen and be superimposed on the first display screen, performs processing for an operation for the transmissive region as an operation for the second display screen, and performs processing for an operation for a region other than the transmissive region as an operation for the first display screen.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a display device and the like. [Background technology]

[0002] 2. Description of the Related Art Conventionally, various devices have been provided with display means for displaying information, and techniques for improving user convenience have also been proposed.

[0003] For example, an information processing device has been proposed that includes a display unit that displays a rear view screen superimposed behind a transparently displayed front view screen, a front touch panel that accepts operations on the front view screen, and a rear touchpad that accepts operations on the rear view screen and is provided independently of the front touch panel (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-111341 Summary of the Invention [Problem to be solved by the invention]

[0005] The user interface (UI) of an information processing device used by multiple users in an office, such as a digital multifunction peripheral (image forming device), often has a single-screen configuration, due to the limited functionality and relatively small screen size of the information processing device. In other words, information processing devices do not output multiple screens using a window system like a personal computer. Even if a window system is adopted, a single window is displayed full-screen. In recent years, devices such as image forming devices have become required to access a network, so they may incorporate a web browser and implement a UI on the web browser. Web browsers can manage and display multiple content. Therefore, even with a single-screen configuration like the UI of an image forming device, user convenience can be improved by simultaneously displaying and operating internal device content (internal content) and external content (content obtained from an external device, such as an external server).

[0006] Here, when using a single screen (full-screen display in one window), internal content and external content are typically composited and displayed using an iframe tag in HTML (HyperText Markup Language). However, due to security constraints such as cross-domain restrictions, it may not be possible to composite and display internal content and external content on a single screen (full-screen display in one window). In other words, when attempting to simultaneously manage and display internal content (such as a copy screen, scan screen, or system area) and external content (cloud services on the Internet) using a web browser installed in an image forming device, the composite display may not be possible on a single screen. To solve this problem, it is possible to display the internal content and external content in separate windows. In this case, there is a problem that it is desirable to be able to input operations similar to those for a single screen configuration, but prior art such as Patent Document 1 does not take this problem into consideration.

[0007] In view of the above-mentioned problems, the present disclosure has an object to provide a display device or the like that is capable of appropriately processing operations when multiple screens are displayed overlapping each other. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the display device of the present disclosure includes a display unit and a control unit, and the control unit displays on the display unit a first display screen that can include a transparent area and a second display screen that is located behind the first display screen and is displayed superimposed on the first display screen, and processes operations on the transparent area as operations on the second display screen, and processes operations on areas other than the transparent area as operations on the first display screen.

[0009] The control method of the present disclosure is a control method for a display device, and is characterized by including the steps of displaying a first display screen that can include a transparent area and a second display screen that is located behind the first display screen and displayed superimposed on the first display screen, and processing operations on the transparent area as operations on the second display screen, and processing operations on areas other than the transparent area as operations on the first display screen. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide a display device or the like that is capable of appropriately processing operations when multiple screens are displayed overlapping each other. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a perspective view showing the appearance of an image forming apparatus according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of the image forming apparatus according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a data configuration of screen setting information in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an outline of processing in the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an outline of processing in the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating an outline of processing in the first embodiment. [Figure 7] FIG. 4 is a flowchart showing the flow of main processing of the image forming apparatus according to the first embodiment. [Figure 8] FIG. 4 is a flowchart showing the flow of processing executed by a browser control unit in the first embodiment. [Figure 9] 4 is a flowchart showing the flow of processing executed by a display control unit in the first embodiment. FIG. [Figure 10] FIG. 3 is a flowchart showing the flow of processing executed by an internal window engine unit in the first embodiment. [Figure 11] FIG. 4 is a diagram illustrating an example of operation in the first embodiment. [Figure 12] FIG. 4 is a diagram illustrating an example of operation in the first embodiment. [Figure 13] FIG. 4 is a diagram illustrating an example of operation in the first embodiment. [Figure 14] FIG. 10 is a diagram illustrating an outline of processing in the second embodiment. [Figure 15] FIG. 10 is a flowchart showing the flow of processing executed by an external window engine unit in the second embodiment. [Figure 16] FIG. 10 is a flowchart showing the flow of processing executed by an external window engine unit in the second embodiment. [Figure 17] FIG. 10 is a flowchart showing the flow of processing executed by a display control unit in the second embodiment. [Figure 18] FIG. 10 is a flowchart showing the flow of processing executed by a browser control unit in the second embodiment. [Figure 19] FIG. 10 is a flowchart showing the flow of processing executed by an internal window engine unit in the second embodiment. [Figure 20] FIG. 10 is a diagram illustrating an example of operation in the second embodiment. [Figure 21] FIG. 10 is a diagram illustrating a functional configuration of an image forming apparatus according to a third embodiment. [Figure 22] FIG. 11 is a diagram illustrating an example of a data configuration of touch information according to the third embodiment. [Figure 23] FIG. 11 is a flowchart showing the flow of processing executed by an internal window engine unit in the third embodiment. [Figure 24] FIG. 10 is a diagram illustrating an example of operation in the third embodiment. [Figure 25] FIG. 10 is a diagram illustrating an example of operation in the third embodiment. [Figure 26] FIG. 10 is a diagram illustrating a functional configuration of an image forming apparatus according to a fourth embodiment. [Figure 27] FIG. 13 is a flowchart showing the flow of processing executed by an internal window engine unit in the fourth embodiment. [Figure 28] FIG. 13 is a flowchart showing the flow of touch information update processing in the fourth embodiment. [Figure 29] FIG. 10 is a diagram illustrating an example of operation in the fourth embodiment. [Figure 30] FIG. 10 is a diagram illustrating an example of operation in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment for carrying out the present disclosure will be described with reference to the drawings. Note that the following embodiment is an example for explaining the present disclosure, and the technical scope of the invention described in the claims is not limited to the following description.

[0013] [1. First embodiment] [1.1 Functional Configuration] A first embodiment will be described with reference to the drawings. Fig. 1 is a perspective view of the appearance of an image forming apparatus 10 according to the first embodiment, and Fig. 2 is a block diagram showing the functional configuration of the image forming apparatus 10.

[0014] Image forming apparatus 10 is an information processing apparatus having a copy function, a scan function, a document print function, a fax function, etc., and is also called an MFP (Multi-Function Printer / Peripheral). As shown in Fig. 2, image forming apparatus 10 includes a control unit 100, an image input unit 120, an image forming unit 130, a display unit 140, an operation unit 150, a storage unit 160, a communication unit 190, and a power supply unit 195 that supplies power to each functional unit of image forming apparatus 10.

[0015] The control unit 100 is a functional unit for controlling the entire image forming apparatus 10. The control unit 100 realizes various functions by reading and executing various programs stored in the storage unit 160, and is configured, for example, by one or more arithmetic units (CPUs (Central Processing Units)). The control unit 100 may also be configured as an SoC (System on a Chip) having multiple functions among those described below.

[0016] Control unit 100 executes programs stored in storage unit 160 to function as image processing unit 102, display control unit 104, internal window engine unit 106, external window engine unit 108, browser control unit 110, and HTTP (HyperText Transfer Protocol) server unit 112. Here, display control unit 104, internal window engine unit 106, and external window engine unit 108 are realized by executing a web browser application 164, which will be described later. Furthermore, browser control unit 110 is realized by executing a browser controller application 166, which will be described later.

[0017] The image processing unit 102 performs various image-related processes. For example, the image processing unit 102 performs sharpening and tone conversion on the image input by the image input unit 120.

[0018] The display control unit 104 displays two windows, an internal content window (hereinafter referred to as the "internal window") which is a first display screen, and an external content window (hereinafter referred to as the "external window") which is a second display screen, on the display unit 140. The display control unit 104 also causes the internal window and the external window to process operations input by the user to the internal window and the external window.

[0019] The internal and external windows render screens based on the processing of the web browser's display engine (HTML (Hyper Text Markup Language) rendering engine).

[0020] The external window is a window that displays content (such as a cloud service) managed by an external device on a network such as the Internet. The internal window is a window (display area) that displays content (internal content) managed and stored within image forming apparatus 10, and a predetermined area can be made transparent. When external content is displayed on the internal window, the area in which the external content is displayed can be made transparent, allowing the content of the external window to be displayed on display unit 140.

[0021] Display control unit 104 displays two windows, the internal window and the external window, overlapping on display unit 140. Display control unit 104 displays the internal window in front of (in front of) the external window, filling the entire display area of ​​display unit 140. Display control unit 140 also displays the external window behind (behind) the internal window, overlapping it. The front-to-back relationship (Z order) between the internal window and the external window is fixed, and the internal window displayed in front and the external window displayed at the back do not switch places.

[0022] Display control unit 104 makes a partial area of ​​the internal window transparent depending on the screen (content) to be displayed. In this embodiment, the transparent area is called a transparent area. When the internal window includes a transparent area, display unit 140 displays a screen in which the display content of the external window is displayed in the transparent area.

[0023] In this embodiment, the internal content includes a system area at the top. The system area is an area where information about the image forming device 10 and content such as buttons for switching between functions to be used are arranged, and its position and scope (height, etc.) are specified in advance. The display control unit 104 displays the system area regardless of whether the internal window includes a transparent area. On the other hand, the external content does not include the system area. Because the external window does not display the system area, it is smaller in size in the vertical direction (Y-axis direction) than the internal window.

[0024] The internal window engine unit 106 interprets HTML and displays the generated screen (content) in the internal window, and executes JavaScript (registered trademark) programs called from the content. That is, the internal window engine unit 106 is an engine (HTML rendering engine) for the internal window. The external window engine unit 108 is an engine (HTML rendering engine) for the external window.

[0025] In this embodiment, the part (engine) that interprets HTML and generates a screen is also referred to as a browser engine layer. In this embodiment, the browser engine layer is described as being divided into two parts: an internal window engine unit 106 for internal windows and an external window engine unit 108, but the browser engine layer may be a common engine for both internal and external windows.

[0026] The web browser of this embodiment is realized by the above-described display control unit 104, internal window engine unit 106, and external window engine unit 108. The processes executed by the display control unit 104, internal window engine unit 106, and external window engine unit 108 will be described later.

[0027] The browser control unit 110 controls the web browser by performing processes such as notifying the web browser of operation details. The browser control unit 110 is capable of HTTP communication (communication via WebSocket) and performs predetermined communication with the internal window engine unit 106. The processes performed by the browser control unit 110 will be described later. In this embodiment, notification includes sending and receiving predetermined information. In this case, the notifying party sends information to the notified party, and the notified party receives the information.

[0028] The HTTP server unit 112 transmits HTML (HyperText Markup Language) data, CSS (Cascading Style Sheets) data, and image data based on the HTTP protocol. When the HTTP server unit 112 receives an HTTP request, it transmits the requested data to the sender (client) that transmitted the HTTP request.

[0029] The image input unit 120 inputs image data to the image forming apparatus 10. For example, the image input unit 120 is configured by a scanning device or the like that can read an image and generate image data. The scanning device converts the image into an electrical signal using an image sensor such as a CCD (Charge Coupled Device) or a CIS (Contact Image Sensor), and generates digital data by quantizing and encoding the electrical signal.

[0030] Image forming unit 130 forms (prints) an image on a recording medium such as recording paper. Image forming unit 130 is configured, for example, by a laser printer that uses an electrophotographic system. Image forming unit 130 includes a paper feed unit 132 and a printing unit 134. Paper feed unit 132 feeds recording paper. Paper feed unit 132 is configured by a paper feed tray, a manual feed tray, etc. Printing unit 134 forms (prints) an image on the surface of the recording paper and ejects the recording paper from an ejection tray.

[0031] The display unit 140 displays various types of information. The display unit 140 is configured by a display device such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or a micro LED display.

[0032] The operation unit 150 accepts operations from a user of the image forming apparatus 10. The operation unit 150 is configured with an input device such as a touch sensor. The touch sensor may detect input using any common detection method, such as a resistive film method, an infrared method, an electromagnetic induction method, or a capacitance method. The image forming apparatus 10 may be equipped with a touch panel in which the display unit 140 and the operation unit 150 are integrally formed.

[0033] The storage unit 160 stores various programs and various data necessary for the operation of the image forming apparatus 10. The storage unit 160 is configured by a storage device such as a solid state drive (SSD) or a hard disk drive (HDD), which is a semiconductor memory.

[0034] The storage unit 160 stores an operating system 162, a web browser application 164, and a browser controller application 166. Furthermore, the storage unit 160 reserves a content data storage area 168 and a screen setting information storage area 170 as storage areas.

[0035] The operating system 162 is software that serves as the foundation for operating the image forming apparatus 10. The operating system 162 is read and executed by the control unit 100 to execute programs, detect operations input via the operation unit 150, and transmit information about the detected operations (event information) to the programs. The operating system 162 may also provide a platform that realizes the execution of programs and the transmission and reception of event information.

[0036] Web browser application 164 is a program that causes control unit 100 to realize the functions of display control unit 104, internal window engine unit 106, and external window engine unit 108. Browser controller application 166 is a program that causes control unit 100 to realize the functions of browser control unit 110.

[0037] The content data storage area 168 stores content data used to display a screen (content within the image forming apparatus 10) displayed in an internal window. The content data is, for example, HTML data, CSS data, image data, etc.

[0038] The screen setting information storage area 170 stores information (screen setting information) on the settings of the screen displayed on the display unit 140. For example, as shown in Fig. 3, the screen setting information includes a screen name (e.g., "login screen") that identifies the screen, a display setting of an internal window (e.g., "display"), a display setting of an external window (e.g., "hide"), and a URL (Uniform Resource Locator, e.g., "http: / / localhost / login") that indicates the source of content acquisition.

[0039] The display setting for the internal window stores either "display" or "partial display." "display" indicates that the internal window does not include a transparent area. "partial display" indicates that the internal window includes a transparent area. In this embodiment, the transparent area is an area that displays external content, and is an area of ​​the internal content other than the system area.

[0040] The display setting for the external window stores either "display" indicating that the external window is to be displayed, or "hide" indicating that the external window is not to be displayed. In the case of "hide," the external window may be displayed with a blank page (about:blank) displayed and in standby mode.

[0041] The communication unit 190 communicates with external devices via a LAN (Local Area Network) or WAN (Wide Area Network). The communication unit 190 is configured, for example, by a communication device or communication module such as a NIC (Network Interface Card) used in a wired / wireless LAN. The communication unit 190 may also communicate with other devices via a telephone line. In this case, the communication unit 190 is configured by an interface (terminal) into which a cable for connecting to a telephone line can be inserted, and transmits and receives images to and from other devices by performing facsimile communication using known standards and protocols such as the G3 / G4 standard.

[0042] [1.2 Processing Overview] [1.2.1 Internal and External Windows] The relationship between the internal window and the external window will be explained with reference to Figure 4. Figure 4(a) (1) shows the internal window. The internal window includes an area that displays the system area (Figure 4(a) (2)) and an area that displays content included outside the system area (Figure 4(a) (3), hereinafter referred to as the "content display area").

[0043] On the other hand, (4) in Figure 4(a) shows the external window. The external window is a window that is the same size as the content display area. The external window is also located in the same position as the content display area of ​​the internal window. The internal window ((1) in Figure 4(a)) is always displayed in front of the external window ((4) in Figure 4(a)), so the external window is hidden by the content display area of ​​the internal window.

[0044] Fig. 4(b) is a diagram showing an example of a display when internal content (for example, an operation screen for the copy function or the scan function) is displayed. The content used to set the copy function or the scan function or to execute a job is displayed in the content display area of ​​the internal window ((5) in Fig. 4(b)).

[0045] 4(c) is a diagram showing an example of a display when external content is displayed. In this case, the content display area of ​​the internal window ((6) in FIG. 4(c)) becomes a transparent area, and the display contents (external content) of the external window located behind the content display area of ​​the internal window are displayed. As a result, the content in the system area and the external content are displayed on display unit 140.

[0046] [1.2.2 Screen transition] Figure 5 shows an example of the transition from each screen to the next screen. When the image forming apparatus 10 is turned on, it displays a login screen ((1) in Figure 5) and authenticates the user. After authenticating the user, the image forming apparatus 10 displays a home screen ((2) in Figure 5). The home screen is a screen that allows the user to select a function (job) to be performed by the image forming apparatus 10.

[0047] Based on a user's operation, the image forming apparatus 10 displays a setting screen ((3) in FIG. 5) and operation screens for various functions from the home screen. The operation screens include an operation screen for the copy function ((4) in FIG. 5), an operation screen for the print hold function ((5) in FIG. 5), an operation screen for the fax function ((6) in FIG. 5), and an operation screen for the scan function ((7) in FIG. 5). These screens are operation screens for functions (native functions) possessed by the image forming apparatus 10 and are internal contents. On the other hand, cloud service 1 ((8) in FIG. 5) and cloud service 2 ((9) in FIG. 5) are screens that display external contents provided by external devices. Cloud services can be registered via the setting screen. Each screen shown in FIG. 5 is displayed based on screen setting information stored in the screen setting information storage area 170.

[0048] [1.2.3 Operational Information Flow] Fig. 6 is a diagram showing a notification route of operation information (event) when an operation such as a touch operation is performed. Fig. 6(a) shows the operating system (hereinafter referred to as "OS"), Fig. 6(b) shows an internal window, Fig. 6(c) shows an external window, Fig. 6(d) shows the browser control unit 110, and Fig. 6(e) shows the display control unit 104. Note that the screens (contents) displayed in the external window and internal window are generated by the internal window engine unit 106 and external window engine unit 108, which are browser engine layers. In addition, the notified event will be described as a touch event related to a touch operation.

[0049] First, the OS notifies the browser control unit 110 of a touch event ((1) in FIG. 6). The browser control unit 110 notifies the web browser of the notified touch event as a touch event for the internal window using inter-process communication ((2) in FIG. 6). The display control unit 104 of the web browser processes the notified touch event as an event for the internal window ((3) in FIG. 6). At this time, if the internal window is displaying external content, it determines whether the notified touch event is an operation on the display portion of the external content (transparent area of ​​the internal window).

[0050] If the internal window determines that the notified touch event is a touch event for external content, it notifies the browser control unit 110 of the touch event using HTTP communication (WebSocket) ((4) in FIG. 6). The browser control unit 110 notifies the web browser of the notified touch event as a touch event for the external window using inter-process communication ((5) in FIG. 6). The display control unit 104 of the web browser processes the notified touch event as an event for the external window ((6) in FIG. 6).

[0051] In addition, if the internal window does not determine in (3) of Fig. 6 that the operation is directed to the external content, it does not execute the process in (4) of Fig. 6. As a result, the processes in (5) and (6) of Fig. 6 are not executed, and the touch event is simply processed as an event directed to the internal window.

[0052] Furthermore, a web browser is realized by the internal window (FIG. 6(b)), external window (FIG. 6(c)), and display control unit 104 (FIG. 6(e)). The web browser communicates with an internal HTTP server (HTTP server unit 112) and an external HTTP server on the Internet, which is an external server, to acquire content. Furthermore, the web browser displays the acquired content in the internal window or external window through processing by display control unit 104.

[0053] [1.3 Processing flow] Next, the flow of processing executed by the image forming apparatus 10 will be described with reference to Fig. 7 to Fig. 10. The processing shown in Fig. 7 to Fig. 10 is executed by the control unit 100 reading out a program stored in the storage unit 160.

[0054] Here, it is assumed that the control unit 100 runs the OS by reading and executing the operating system 162. Therefore, the control unit 100 detects an operation input by the user (for example, a touch operation input via the operation unit 150). It is also assumed that the control unit 100 causes the display control unit 104, the internal window engine unit 106, the external window engine unit 108, the browser control unit 110, and the HTTP server unit 112 to function in the OS. When the OS run by the control unit 100 detects an operation input by the user, it notifies the browser control unit 110 of the fact that the operation has been performed (an event), and also notifies the browser control unit 110 of information indicating the content of the operation.

[0055] [1.3.1 Main Processing] The flow of main processing executed by image forming apparatus 10 of the present embodiment will be described with reference to Fig. 7. The processing shown in Fig. 7 is executed when the screen displayed on display unit 140 is updated.

[0056] First, the control unit 100 reads out screen setting information for a screen to be displayed on the display unit 140 from the screen setting information storage area 170 based on the user's operation and the state of the image forming apparatus 10 (step S100).

[0057] Next, control unit 100 applies the display settings of the internal window included in the screen setting information read in step S100 to the internal window (step S102). Also, control unit 100 applies the display settings of the external window included in the screen setting information read in step S100 to the external window (step S104).

[0058] Next, the control unit 100 displays the content (step S106). For example, if the URL included in the screen setting information read in step S100 includes the domain name of the HTTP server unit 112 (e.g., localhost), the control unit 100 displays the content specified by the URL in the internal window. Furthermore, if the URL included in the screen setting information read in step S100 includes a domain name other than the domain name of the HTTP server unit 112, the control unit 100 displays the content specified by the URL in the external window.

[0059] [1.3.2 Browser Control Processing] The process executed by browser control unit 110 will be described with reference to Fig. 8. Browser control unit 110 repeatedly executes the process shown in Fig. 8.

[0060] First, browser control unit 110 determines whether a touch event has been notified from the OS (step S120). The touch event is notified together with information (operation information) indicating the content of the operation, such as the touched position and the status of the touch operation. The information on the status of the touch operation is information on the operation of the touch operation, such as whether a new touch position has been set (the touch operation has started), whether the touch position has moved, or whether the touch position has been removed (the touch operation has ended).

[0061] When the browser control unit 110 is notified of a touch event from the OS, it notifies the browser (display control unit 104) of the touch event as a touch event for an internal window using inter-process communication (step S120; Yes→step S122).

[0062] On the other hand, if no touch event is notified from the OS, the browser control unit 110 determines whether or not a touch event for an external window has been notified from the internal window (step S120; No→step S124). In this embodiment, the touch event for an external window is notified to the browser control unit 110 from the internal window engine unit 106 using HTTP communication (WebSocket). If a touch event for an external window is notified, the browser control unit 110 notifies the browser (display control unit 104) of the touch event for the external window using inter-process communication (step S124; Yes→step S126). As a result, the browser control unit 110 notifies the browser (display control unit 104) of the touch event notified in step S122, this time as a touch event for the external window. If no touch event for the external window is notified, the browser control unit 110 omits the processing in step S126 (step S124; No).

[0063] [1.3.3 Display control section] The process executed by the display control unit 104 will be described with reference to Fig. 9. The display control unit 104 repeatedly executes the process shown in Fig. 9.

[0064] First, display control unit 104 determines whether or not a touch event for an internal window has been notified from browser control unit 110 (step S130). If a touch event for an internal window has been notified, display control unit 104 processes the touch event as a touch event for an internal window (step S130; Yes→step S132). For example, display control unit 104 notifies internal window engine unit 106 (browser engine layer) of the touch event.

[0065] On the other hand, if a touch event for an internal window has not been notified, display control unit 104 determines whether a touch event for an external window has been notified from browser control unit 110 (step S130; No→step S134). If a touch event for an external window has been notified, display control unit 104 processes the touch event as a touch event for the external window (step S134; Yes→step S136). For example, display control unit 104 notifies external window engine unit 108 (browser engine layer) of the touch event. Note that if a touch event for an external window has not been notified, browser control unit 110 omits the processing in step S136 (step S134; No).

[0066] [1.3.4 Internal Window Engine] The process executed by the internal window engine unit 106 will be described with reference to Fig. 10. The internal window engine unit 106 repeatedly executes the process shown in Fig. 10.

[0067] First, the internal window engine unit 106 determines whether or not a touch event has been notified from the display control unit 104 (step S140). If the internal window engine unit 106 determines that a touch event has not been notified, it repeats the process of step S140 (step S140; No).

[0068] On the other hand, when the internal window engine unit 106 is notified of a touch event, it determines whether or not a touch operation has been performed on a transparent region based on the operation information notified together with the touch event (step S140; Yes→step S142). If the touch operation is not on a transparent region, the internal window engine unit 106 processes the touch operation as a touch operation on an internal window (step S142; No→step S144). On the other hand, if the touch operation is on a transparent region, the internal window engine unit 106 notifies the browser control unit 110 of the touch event notified in step S140 as a touch event for an external window using HTTP communication (WebSocket) (step S142; Yes→step S146).

[0069] [1.3.5 External Window Engine] When the external window engine unit 108 is notified of a touch event for an external window from the display control unit 104, it processes the touch operation based on the touch event.

[0070] 7 to 10, an operation on a transparent area is notified to the external window engine unit 108 as a touch event for the external window, and is processed as an operation on the external window. An operation on an area other than the transparent area is processed by the internal window engine unit 106 as an operation on the internal window.

[0071] [1.4 Example of operation] An example of operation in this embodiment will be described. Fig. 11(a) is an example of a login screen W100. The login screen W100 is displayed when user authentication is enabled on the setting screen of the image forming apparatus 10. The login screen W100 includes a system area E100 and a content display area E102. The content of the login screen is displayed in the content display area E102. The content of the login screen includes a login name input field T100, a password input field T102, a button B100 for logging in, and the like.

[0072] 11(b) is an example of the display on the display screen W110 when the login name input field T100 or the password input field T102 is touched. When the input field is touched, the image forming apparatus 10 of this embodiment displays a software keyboard in an internal window. The software keyboard function is generally provided by an OS or platform, but may not be provided in the case of an embedded device such as the image forming apparatus 10. In this case, the image forming apparatus 10 realizes the software keyboard function in the internal window using HTML and JavaScript.

[0073] 11(c) is an example of the display screen W120 when the dialog E120 is displayed. The dialog E120 is displayed when notifying the user of certain information, such as when the password entered on the login screen W100 is incorrect. If the OS or platform does not provide a function for displaying a dialog, the image forming apparatus 10 realizes the function for displaying a dialog in an internal window using HTML and JavaScript.

[0074] In this specification, the software keyboard and dialog are referred to as a native GUI (Graphical User Interface). The native GUI is a component (GUI, UI (User Interface) part) that serves as an input object for allowing a user to perform a predetermined input operation, such as selecting a button or entering a character string. The image forming apparatus 10 realizes an input function by implementing (displaying) a component (input object) having the same function as the native GUI using an internal window. In the following description, a component (input object) that realizes the same function as the native GUI displayed in the internal window will simply be referred to as a native GUI.

[0075] FIG. 12(a) is an example of a home screen W130, which is an initial screen displayed when login is successful or when user authentication is disabled on the settings screen. The home screen includes, for example, an area E130 including function buttons. The function buttons are buttons for selecting functions to be executed by the image forming apparatus 10. The area E130 includes, for example, four function buttons: a copy function button B130, a print hold function button B131, a fax function button B132, and a scan function button B133. The home screen W130 also includes a button B134 for displaying a settings screen, a button B135 for adjusting the volume, a button B136 for adjusting the brightness of the display unit 140, and the like.

[0076] The home screen W130 allows you to change the layout of or add function buttons via a settings screen. If it is not possible to display all function buttons on one screen, area E130 of the home screen W130 can be scrolled left and right by selecting the triangle buttons (button B137 and button B138) or by flicking or scrolling.

[0077] FIG. 12(b) is an example of a display screen W140 that is displayed when button B136 on home screen W130 is selected. Display screen W140 includes a pop-up window E140 for adjusting brightness. FIG. 12(c) is an example of a display screen W150 that is displayed when button B135 on home screen W130 is selected. Display screen W150 includes a pop-up window E150 for adjusting volume. Pop-up window E140 and pop-up window E150 are implemented using HTML and JavaScript.

[0078] Fig. 13(a) is an example of a home screen W160 when button B138, a right-facing triangle button, on home screen W130 shown in Fig. 12(a) is selected and area E130 of home screen W130 is scrolled to the right. Home screen W160 includes, as function buttons, function button B160, which is a button for displaying content from cloud service 1, and function button B161, which is a button for displaying content from cloud service 2. The user can use cloud services (external content) by selecting function button B160 or function button B161.

[0079] Fig. 13(b) is an example of an operation screen W170 that is displayed when the copy function button B130 is selected from the home screen W130 shown in Fig. 12(a). The copy function is a native function provided in the image forming apparatus 10. The operation screen W170 for the copy function is internal content and is displayed in an internal window.

[0080] FIG. 13(c) is an example of an operation screen W180 that is displayed when the cloud service 1 function button B160 is selected from the home screen W160 shown in FIG. 13(a), and is an example of a screen when an authentication screen for cloud service 1 is displayed. Here, a system area E180, which is the upper area of ​​the internal window, is displayed without being transparent. The system area includes, for example, a home button B180 for switching the operation screen W180 to the home screen. Furthermore, a content area E181, which is the lower area of ​​the internal window, is a transparent area. Therefore, a screen of an external cloud service is displayed in the content area E181.

[0081] The user can perform a touch operation on the operation screen W180. Here, when the user performs an operation of touching an area other than the transparent area of ​​the internal window (the system area E180), the operation is processed as a touch operation on the internal window. Therefore, when the user touches the home button B180 included in the system area E180, the image forming apparatus 10 determines that the home button B180 has been touched and switches the operation screen W180 to the home screen. On the other hand, when the user performs an operation of touching the transparent content area E181 (transparent area), the operation is processed by the image forming apparatus 10 as a touch operation on the external window.

[0082] In the above-described embodiment, the process of notifying a touch operation (touch event) has been described, but a mouse operation (mouse event) may also be notified in a similar process.

[0083] In this way, the image forming apparatus of this embodiment has a two-window configuration consisting of an inner window and an outer window, but allows the user to perform touch operations as if it were a single-screen configuration.

[0084] The image forming apparatus of this embodiment displays the external content in an external window separate from the internal window displaying the internal content, thereby enabling the image forming apparatus of this embodiment to handle cases where the internal content and the external content cannot be displayed using iframe tags due to cross-domain restrictions.

[0085] Generally, to avoid cross-domain restrictions, an external HTTP server must be configured to allow cross-domain access. However, this increases the management burden on the external content side (external HTTP server) and is ineffective when changing settings on the external HTTP server (cloud service). In particular, the external HTTP server may enforce cross-domain restrictions to prevent clickjacking when content is displayed using an iframe tag, potentially reducing security by changing the settings. The image forming apparatus of this embodiment solves these problems by providing a two-window UI on the image forming apparatus side (client side) without changing settings on the external HTTP server side. Furthermore, the image forming apparatus of this embodiment allows users to perform touch operations similar to those of a single-screen configuration, improving user convenience. The image forming apparatus of this embodiment, despite its two-window configuration, eliminates the need for touch operations to switch between windows, allowing users to perform seamless touch operations without compromising the usability of a single-window configuration.

[0086] [2. Second Embodiment] Next, a second embodiment will be described. In addition to the processing described in the first embodiment, the second embodiment is an embodiment in which processing is executed to realize a native GUI for an external window based on an operation on the external window.

[0087] In the first embodiment, the native GUI is displayed in an internal window. However, there are cases where the native GUI cannot be displayed in an external window (single window). This is due to restrictions on iframes, for example, a software keyboard, which is internal content, cannot be displayed in a web browser (external window) that displays external content. In this way, there are cases where the native GUI, which should ideally be displayed in the same window, cannot be displayed in the same window.

[0088] Therefore, the image forming apparatus 10 of this embodiment realizes a native GUI in a dedicated window (internal window) that ensures security, and makes the native GUI available from an external window, thereby realizing a native GUI using a browser while ensuring security. This allows the image forming apparatus 10 to allow the user to perform input operations on external content, and makes it possible for the content input by the user to be reflected in the external content.

[0089] In this embodiment, the native GUI realized in the internal window is as follows. (1) Software keyboard A software keyboard is a keyboard implemented by software that displays the keys typically found on a keyboard, as well as an [OK] button and a [Cancel] button. When the user selects the [OK] button, the input content (character strings) entered using each key is reflected in the content displayed in the internal or external window. (2) Dialogue A dialog is a window (dialog box) that is displayed to display information or to request the user to select a button or input information. In this embodiment, the following four types of dialogs are displayed: (2-1) JavaScript alert dialog A JavaScript alert dialog is a dialog that includes a message and an [OK] button. It is displayed when a process for displaying the alert dialog is executed in a JavaScript program. (2-2) JavaScript confirm dialog The JavaScript confirm dialog is a dialog that includes a message, an [OK] button, and a [Cancel] button. It is displayed when a process for displaying the confirm dialog is executed in a JavaScript program. (2-3) JavaScript prompt dialog A JavaScript prompt dialog is a dialog that includes a message, a text input field, an OK button, and a Cancel button. It is displayed when a JavaScript program executes a process to display the prompt dialog. (2-4) Authentication dialog The authentication dialog is a dialog that is displayed when HTTP401 (authentication failed, an HTTP response with an HTTP response code of 401) is returned by the content server. The authentication dialog is a dialog that includes two input fields for entering authentication information: a text input field for entering an account name and a text input field for entering a password, and also includes an [OK] button and a [Cancel] button.

[0090] In this embodiment, a JavaScript alert dialog, a JavaScript confirm dialog, and a JavaScript prompt dialog are referred to as a JavaScript dialog.

[0091] [2.1 Information Flow] 14 is a diagram showing a notification route of an event (information) inside the image forming apparatus 10 when an operation to call a native GUI from an external window (an operation to display a software keyboard or a process to display a dialog) is performed. Note that (a) to (e) in FIG. 14 are the same functional units as (a) to (e) in FIG. 6.

[0092] First, the web browser (external window) detects that an operation or process to display a native GUI has been performed. At this time, the external window engine unit 108 notifies the display control unit 104 of a request to display the native GUI (native GUI activation request) ((1) in FIG. 14). The display control unit 104 notifies the browser control unit 110 of the native GUI activation request ((2) in FIG. 14). The browser control unit 110 notifies the internal window engine unit 106 of the native GUI activation request using HTTP communication (WebSocket) ((3) in FIG. 14). Upon receiving the native GUI activation request, the internal window engine unit 106 displays the native GUI in the internal window.

[0093] After the user completes the native GUI operation, the internal window engine unit 106 notifies the browser control unit 110 that the native GUI operation has ended (the result of the native GUI operation) using HTTP communication (WebSocket) ((4) in FIG. 14). The browser control unit 110 notifies the browser (display control unit 104) that the native GUI operation has ended (the operation result) using inter-process communication ((5) in FIG. 14). The web browser (display control unit 104) reflects the operation result in the external content ((6) in FIG. 14).

[0094] [2.2 Processing flow] Next, the flow of processing executed by image forming apparatus 10 will be described with reference to Figures 15 to 19. The processing shown in Figures 15 to 19 is executed by control unit 100 reading out a program stored in storage unit 160. The processing shown in Figures 15 to 19 is executed in parallel with the processing shown in Figures 7 to 10 of the first embodiment.

[0095] [2.2.1 External window engine (determination process)] The determination process executed by the external window engine unit 108 will be described with reference to Fig. 15. The determination process is a process for determining whether or not an operation or process for displaying a native GUI has been performed. The external window engine unit 108 repeatedly executes the process shown in Fig. 15.

[0096] First, the external window engine unit 108 determines whether or not authentication has failed when loading a page (when acquiring content) (step S200). For example, the external window engine unit 108 determines that authentication has failed when an HTTP response with an HTTP response code of 401 is returned from the external HTTP server. If authentication has failed, the external window engine unit 108 notifies the display control unit 104 of a request to start a native GUI of an authentication dialog (step S200; Yes→step S202).

[0097] On the other hand, if the authentication at the time of page loading did not fail, the external window engine unit 108 determines whether or not there has been a native GUI startup request for the JavaScript dialog (step S200; No→step S204). A native GUI startup request for the JavaScript dialog is a request for displaying an alert dialog, a confirm dialog, and a prompt dialog, which is made when processing for displaying these dialogs is executed in a JavaScript program. If there has been a native GUI startup request for the JavaScript dialog, the external window engine unit 108 notifies the display control unit 104 of the native GUI startup request for the JavaScript dialog (step S204; Yes→step S206).

[0098] On the other hand, if there is no request for starting a native GUI of the JavaScript dialog, the external window engine unit 108 determines whether or not a character input operation has been performed (step S204; No→step S208). For example, the external window engine unit 108 determines that a character input operation has been performed when an operation of touching a character string input field displayed by an input tag or a textarea tag has been performed. If there is a character input operation, the external window engine unit 108 notifies the display control unit 104 of a request for starting a native GUI of the software keyboard (step S208; Yes→step S210). Note that if there is no character input operation, the external window engine unit 108 omits the processing in step S210 (step S208; No).

[0099] [2.2.2 External window engine (result reflection processing)] The result reflection process executed by the external window engine unit 108 will be described with reference to Fig. 16. The result reflection process is a process for reflecting a result response (operation result) to the native GUI in an external window. The external window engine unit 108 repeatedly executes the process shown in Fig. 16.

[0100] First, the external window engine unit 108 determines whether or not a result response to the native GUI of the authentication dialog has been notified (step S220). The result response to the native GUI of the authentication dialog is, for example, information including the account name and password entered via the authentication dialog. When a result response to the native GUI of the authentication dialog has been notified, the external window engine unit 108 notifies the external HTTP server of the result (the entered account name and password) (step S220; Yes→step S222). Note that, when authentication by the external HTTP server is successful, the display control unit 104 and the external window engine unit 108 continue processing to acquire content from the external HTTP server and display the acquired content.

[0101] On the other hand, if the result response for the native GUI of the authentication dialog has not been notified, the external window engine unit 108 determines whether the result response for the native GUI of the JavaScript dialog has been notified (step S220; No→step S224). The result response for the native GUI of the JavaScript dialog is information including, for example, information indicating the selected button and information on the input character string. If the result response for the native GUI of the JavaScript dialog has been notified, the external window engine unit 108 reflects the button selected by the user and the input character string in the external content (step S224; Yes→step S226).

[0102] On the other hand, if the result response for the native GUI of the JavaScript dialog has not been notified, the external window engine unit 108 determines whether the result response for the native GUI of the software keyboard has been notified (step S224; No→step S228). The result response for the native GUI of the software keyboard is, for example, information including information on the character string entered by the user. If the result response for the native GUI of the software keyboard has been notified, the external window engine unit 108 reflects the character string entered by the user in the character string input field selected in step S208 of FIG. 15 (step S228; Yes→step S230). Note that if the result response for the native GUI of the software keyboard has not been notified, the external window engine unit 108 omits the processing in step S230 (step S228; No).

[0103] [2.2.3 Display control section] The process executed by the display control unit 104 will be described with reference to Fig. 17. The display control unit 104 repeatedly executes the process shown in Fig. 17.

[0104] First, display control unit 104 determines whether or not a native GUI startup request has been notified from external window engine unit 108 (step S250). If a native GUI startup request has been notified, display control unit 104 notifies browser control unit 110 of the native GUI startup request using inter-process communication (step S250; Yes→step S252).

[0105] On the other hand, if the native GUI activation request has not been notified, the display control unit 104 determines whether or not a native GUI result response has been notified from the browser control unit 110 (step S250; No→step S254). If a result response has been notified, the display control unit 104 notifies the notified result response to the external window engine unit 108 (step S254; Yes→step S256). Note that if a native GUI result response has not been notified, the display control unit 104 omits the processing in step S256 (step S254; No).

[0106] [2.2.4 Browser Control] The process executed by browser control unit 110 will be described with reference to Fig. 18. Browser control unit 110 repeatedly executes the process shown in Fig. 18.

[0107] First, browser control unit 110 determines whether or not a native GUI startup request has been notified from display control unit 104 (step S260). If a native GUI startup request has been notified, browser control unit 110 notifies internal window engine unit 106 of the native GUI startup request using HTTP communication (WebSocket) (step S260; Yes→step S262).

[0108] On the other hand, if the browser control unit 110 has not received a native GUI activation request, it determines whether or not a native GUI result response has been received from the internal window engine unit 106 (step S260; No→step S264). If the native GUI result response has been received, the browser control unit 110 notifies the web browser (display control unit 104) of the received result response using inter-process communication (step S264; Yes→step S266). If the native GUI result response has not been received, the browser control unit 110 omits the processing in step S266 (step S264; No).

[0109] 2.2.5 Internal Window Engine The process executed by the internal window engine unit 106 will be described with reference to Fig. 19. The internal window engine unit 106 repeatedly executes the process shown in Fig. 19.

[0110] First, the internal window engine unit 106 determines whether or not a native GUI activation request for the authentication dialog has been notified from the browser control unit 110 (step S280). If a native GUI activation request for the authentication dialog has been notified, the internal window engine unit 106 displays the authentication dialog in an internal window (step S280; Yes → step S282). At this time, the internal window engine unit 106 makes the system area and the area other than the area where the authentication dialog is displayed transparent. As a result, the authentication dialog is displayed superimposed on the external content.

[0111] When the operation on the authentication dialog is completed, the internal window engine unit 106 notifies the browser control unit 110 of a result response using HTTP communication (WebSocket) (step S284). For example, when the user selects the [OK] button, the internal window engine unit 106 notifies the browser control unit 110 of a result response including the account name and password entered by the user. Furthermore, when the user selects the [Cancel] button, the internal window engine unit 106 notifies the browser control unit 110 of a result response including information indicating that the [Cancel] button was selected.

[0112] On the other hand, if the internal window engine unit 106 has not received a native GUI startup request for the authentication dialog, it determines whether or not the browser control unit 110 has notified it of a native GUI startup request for a JavaScript dialog (step S280; No→step S286). If the internal window engine unit 106 has received a native GUI startup request for a JavaScript dialog, it displays the requested type of JavaScript dialog in the internal window (step S286; Yes→step S288). At this time, the internal window engine unit 106 makes the system area and the area other than the area where the JavaScript dialog is displayed transparent.

[0113] When the operation on the JavaScript dialog is completed, the internal window engine unit 106 notifies the browser control unit 110 of a result response using HTTP communication (WebSocket) (step S290). For example, the internal window engine unit 106 notifies the browser control unit 110 of a result response including information indicating the button selected by the user and information on the character string entered by the user.

[0114] On the other hand, if the internal window engine unit 106 has not received a native GUI startup request for the JavaScript dialog, it determines whether or not the browser control unit 110 has notified it of a native GUI startup request for the software keyboard (step S286; No→step S292). If the internal window engine unit 106 has notified it of a native GUI startup request for the software keyboard, it displays the software keyboard in the internal window (step S292; Yes→step S294). At this time, the internal window engine unit 106 makes the system area and the area other than the area where the software keyboard is displayed transparent.

[0115] When the operation on the software keyboard is completed, the internal window engine unit 106 notifies the browser control unit 110 of a result response using HTTP communication (WebSocket) (step S296). For example, when the [OK] button is selected by the user, the internal window engine unit 106 notifies the browser control unit 110 of a result response including the character string entered by the user and information indicating that the [OK] button has been selected. Furthermore, when the [Cancel] button is selected by the user, the internal window engine unit 106 notifies the browser control unit 110 of a result response including information indicating that the [Cancel] button has been selected. Note that when a request to start a native GUI of the software keyboard has not been notified, the internal window engine unit 106 omits the processes in steps S294 and S296 (step S292; No).

[0116] [2.3 Example of operation] An example of operation in this embodiment will be described with reference to Fig. 20. Fig. 20(a) is an example of a display screen W200 in which a software keyboard E200 is displayed in an internal window. The software keyboard E200, which is a native GUI, is displayed in the internal window when a character string input field for an account name (ID), password, etc. is touched in cloud service content displayed in an external window. The internal window displays a system area and software keyboard at the top, and the other areas are transparent. As a result, the software keyboard is displayed superimposed on the external content.

[0117] FIG. 20(b) is an example of a display screen W210 in which a JavaScript dialog E210 is displayed in an internal window. The JavaScript dialog E210 is displayed, for example, when the password entered by the user is incorrect. The JavaScript dialog E210 is displayed in an internal window, similar to a software keyboard. FIG. 20(b) is an example in which an alert dialog containing the message "The password is incorrect" is displayed.

[0118] In the above-described embodiment, the native GUI is described as a software keyboard and a dialog box, but the native GUI may be something other than a software keyboard and a dialog box as long as it allows the user to perform input operations for external content. For example, the image forming apparatus 10 may display, as the native GUI, a screen for selecting a date or a time and date, or a screen for inputting an email address or a URL (Uniform Resource Locator).

[0119] In this way, the image forming apparatus of this embodiment can appropriately display the native GUI and reflect operations on the native GUI even when the native GUI is not provided by the operating system.

[0120] 3. Third Embodiment Next, a third embodiment will be described. In the third embodiment, in addition to the processing described in the first embodiment, a browser engine layer (internal window engine unit) executes processing for managing multi-touch operations. In this embodiment, FIG. 2 of the first embodiment is replaced with FIG. 21, and FIG. 10 of the first embodiment is replaced with FIG. 23. Note that the same functional units and processing are assigned the same reference numerals, and their description will be omitted.

[0121] In this embodiment, in a two-window configuration consisting of an internal window and an external window, when a first touch is initiated and that first touch or multiple touches are made, all of those touch operations are treated as a continuous touch operation and as touch operations in the window where the first touch was made until all touch operations are completed.

[0122] In this embodiment, when a touch operation is performed across windows, that is, when a first touch is made and then a window different from the window on which the first touch was made is touched, the touch operation is treated as processing for the window on which the first touch started. In other words, while multiple touch operations are being processed as a series of touch operations, the series of touch operations is always processed as a touch operation on either the internal window or the external window.

[0123] [3.1 Functional Configuration] The functional configuration of the image forming apparatus 12 in this embodiment will be described with reference to Fig. 21. Compared to the image forming apparatus 10 shown in Fig. 2, the image forming apparatus 12 further stores a touch information management table 172 and window information 174 in the storage unit 160.

[0124] The touch information management table 172 is a table used to manage (store) information on touch operations. As shown in Fig. 22, the touch information management table 172 stores, in association with each other, a touch number (e.g., "1") that identifies touch information, a touch ID (e.g., "1") that is a unique number that identifies a point of contact on the touch surface (operation unit 150), whether or not there is a touch (e.g., "present"), an X coordinate (e.g., "600.0") and a Y coordinate (e.g., "200.0") that indicate the touched coordinates, and a touch operation (e.g., "start"), for example.

[0125] The touch ID is, for example, a touch ID acquired by a JavaScript touch operation event handler. The coordinates are expressed as (x, y) where the pixel at the top left corner of the display unit 140 is the origin (0, 0), x is the number of horizontal pixels from the origin to the pixel of interest, and y is the number of vertical pixels. For example, the X coordinate of the touch information management table 172 stores values ​​from 0 to 639, and the Y coordinate stores values ​​from 0 to 479. The operation stores one of the values ​​"start," "move," and "end." "start" indicates that a new touch position has been set (the touch operation has started). "move" indicates that the touch position has moved. "end" indicates that the touch position has been removed (the touch operation has ended). The initial value of the operation is "end."

[0126] In this embodiment, the operation unit 150 is a touch panel that allows up to five touch points, and if six or more touch points are touched, touch events for the sixth point and beyond are not notified. Therefore, touch operation information is managed up to five points, and the touch number is any value from 1 to 5.

[0127] The window information 174 is information indicating the window on which the first touch was initiated. The initial value of the window information 174 is NULL, and when the first touch is initiated, information on either the "internal window" or the "external window" is stored. When all touch operations are completed, the window information 174 stores NULL.

[0128] [3.2 Processing flow] With reference to FIG. 23, the processing executed by the internal window engine unit 106 in this embodiment will be described. First, when a touched window is notified, the internal window engine unit 106 determines whether the window information 174 is NULL (step S300). If the window information 174 is NULL, the internal window engine unit 106 sets information indicating the touched window in the window information 174 (step S300; Yes → step S302). For example, if a transparent portion of an internal window is touched, the internal window engine unit 106 stores "external window" in the window information 174, and otherwise stores "internal window" in the window information 174. Note that if the window information 174 is not NULL, the internal window engine unit 106 omits the processing in step S302 (step S300; No).

[0129] Next, the internal window engine unit 106 determines whether to update the touch information managed in the touch information management table 172 (step S304). If the touch operation corresponds to "move" or "end", the internal window engine unit 106 determines to update the touch information. On the other hand, if the touch operation corresponds to "start", the internal window engine unit 106 determines not to update the touch information (to add touch information).

[0130] If the touch information is not to be updated, the internal window engine unit 106 changes the variable n for the touch number from 1 to the maximum value of the touch number (5 in this embodiment) (step S306). The internal window engine unit 106 refers to the touch information management table 172 and determines whether the touch presence / absence stored in the touch information whose touch number is the variable n is "absent" (step S308). If the touch presence / absence is "absent," the internal window engine unit 106 stores the touch ID, coordinates, and action based on the touch event notified in step S140 in the touch information whose touch number is the variable n, and sets the touch presence / absence to "present." As a result, the internal window engine unit 106 adds the touch information to the touch information management table 172 (step S310).

[0131] On the other hand, when updating touch information (step S304; Yes), the internal window engine unit 106 acquires a touch ID based on the touch event notified in step S140. Then, the internal window engine unit 106 updates the touch information (touch information to be updated) in which the touch ID is stored based on the touch event notified in step S140 (step S312). Here, if the touch operation is an operation corresponding to "end", the internal window engine unit 106 stores "0.0" in the X coordinate and Y coordinate of the touch information to be updated and sets the touch presence / absence to "absence", thereby initializing (clearing) the touch information.

[0132] Next, the internal window engine unit 106 determines whether an "external window" is stored in the window information 174 (step S314). If an "external window" is not stored in the window information 174, the internal window engine unit 106 processes the operation based on the touch information stored in the touch information management table 172 as a touch operation on the internal window (step S314; No→step S144). On the other hand, if the window stored in the window information 174 is an "external window," the internal window engine unit 106 notifies the browser control unit 110 of the operation (touch event) based on the touch information stored in the touch information management table 172 as a touch event for the external window (step S314; Yes→step S316). At this time, the internal window engine unit 106 subtracts a value corresponding to the height of the system area from the Y coordinate information and notifies the browser control unit 110 of the result.

[0133] Next, the internal window engine unit 106 determines whether or not all of the actions of the touch information stored in the touch information management table 172 are "end" (step S318). If all of the actions of the touch information are "end", the internal window engine unit 106 sets NULL to the window information 174 (step S318; Yes -> step S320). If all of the actions of the touch information are not "end", the internal window engine unit 106 omits the processing in step S320 (step S318; No).

[0134] In this way, the internal window engine unit 106 regards other touch operations input from the start of the first touch operation until the end of the first touch operation and touch operations input in conjunction with the other touch operations as touch operations on the window on which the first touch operation was made. As a result, the internal window engine unit 106 can process the series of touch operations as operations on the window corresponding to the first touch position.

[0135] For example, after a touch operation on a transparent region (external window) is initiated, another touch operation may be performed before the touch operation is completed. In this case, the internal window engine unit 106 notifies the display control unit 104 of information (touch event) about the other touch operation and a touch operation input before all of the other touch operations are completed (a touch operation input in conjunction with another touch operation). As a result, if another touch operation is performed after a touch operation on a transparent region (external window) is initiated, the internal window engine unit 106 can process the touch operation input before all of the touch operations are completed as an operation on the external window. Similarly, if a touch operation on a region other than a transparent region (internal window) is initiated, the internal window engine unit 106 processes the touch operation input before all of the touch operations are completed as a touch operation on the internal window if another touch operation is performed after the touch operation is initiated.

[0136] [3.3 Example of operation] An example of operation in this embodiment will be described with reference to Fig. 24 and Fig. 25. Fig. 24 and Fig. 25 are diagrams showing a display screen W300 including an area E300 where an internal window is displayed and an area E302 (transparent area of ​​the internal window) where an external window is displayed, T300 showing the contents stored in touch information management table 172, and D300 showing the contents stored in window information 174. Note that T300 indicates, from left to right, the touch number, whether or not there is a touch, the X coordinate, the Y coordinate, and the operation, and the numbers included in the display screen W300 correspond to the touch numbers.

[0137] 24(a) is a diagram showing a case where no touch operation is performed. When no touch operation is performed, the touch information stored in the touch information management table 172 is cleared, and NULL is stored in the window information 174.

[0138] Fig. 24(b) is a diagram showing a case where a first touch operation is performed on an external window. As shown in T300 of Fig. 24(b), first touch information (M310) is added to the touch information management table 172. Furthermore, as shown in D300 of Fig. 24(b), "external window" is stored in the window information 174.

[0139] Fig. 24(c) is a diagram showing a case where a second touch operation is newly performed while a first touch operation is being performed. As shown in T300 of Fig. 24(c), second touch information (M320) is added to the touch information management table 172. Meanwhile, as shown in D300 of Fig. 24(c), the "external window" remains stored in the window information 174. In this case, the first touch operation and the second touch operation are processed as touch operations on the external window.

[0140] 24(d) is a diagram showing a case where the touch position touched in the first touch operation is moved to the area where the internal content is displayed (internal window). As shown in T300 of FIG. 24(d), the first touch information (M330) in the touch information management table 172 is updated, and the coordinates of the touch position after the movement and the action ("move") are stored in the touch information.

[0141] 25(a) is a diagram showing a case where the touch position touched in the second touch operation is moved to an area where external content is displayed (external window). As shown in T300 of FIG. 25(a), the second touch information (M340) in the touch information management table 172 is updated, and the coordinates of the touch position after the movement and the action ("move") are stored in the touch information.

[0142] It should be noted that the touch operation based on the touch information in FIG. 24(d) and the touch operation based on the touch information in FIG. 25(a) are both processed as touch operations on the external window.

[0143] Figure 25(b) is a diagram showing the case where all touch operations have been completed. The first touch information (M350) and the second touch information (M352) are cleared, and the state returns to the same as in Figure 24(a). If a new touch operation is performed at this time, as shown in Figure 25(c), the touch information management table 172 stores the first touch information (M360) and the window touched at the first point (the "internal window" in the example of Figure 25(c)).

[0144] When the window information 174 is an "internal window," the internal window engine unit 106 processes the touch operation based on the touch information stored in the touch information management table 172. On the other hand, when the window information 174 is an "external window," the internal window engine unit 106 notifies the browser control unit 110 of the touch information stored in the touch information management table 172. The touch information is notified from the browser control unit 110 to the external window engine unit 108 via the display control unit 104, and the external window engine unit 108 processes the touch operation based on the notified touch information.

[0145] In addition, if a touch operation is started and then ends across windows, the internal window engine unit 106 may consider that a drag-and-drop has been performed and pass the information that was selected when the touch operation was started to another window.

[0146] In this way, when a multi-touch operation is performed, the image forming apparatus of this embodiment can process a series of touch operations input from the start of touching until all touch operations are completed as operations on the window corresponding to the first touch position. As a result, even if the touch position straddles windows due to, for example, a swipe operation or a pinch-out operation, the image forming apparatus of this embodiment can process the operations as operations on the window corresponding to the position where the touch operation was started.

[0147] [4. Fourth Embodiment] Next, a fourth embodiment will be described. In the fourth embodiment, multi-touch operations are managed by a method different from that of the third embodiment. In this embodiment, FIG. 2 of the first embodiment is replaced with FIG. 26, and FIG. 10 of the first embodiment is replaced with FIG. 27. Note that the same functional units and processes are assigned the same reference numerals, and their description will be omitted.

[0148] In this embodiment, when a touch operation is continued across windows, the touch operation that was performed before the crossing of the window is treated as a touch end, and the touch operation after the crossing of the window is treated as a touch start in the window that was being touched. In other words, in this embodiment, the touch on each window is managed as a process for each window as it is.

[0149] [4.1 Functional Configuration] The functional configuration of the image forming device 14 in this embodiment will be described with reference to Fig. 26. Compared to the image forming device 10 shown in Fig. 2, the image forming device 14 further stores an internal window touch information management table 176 and an external window touch information management table 178 in the storage unit 160. The information stored in the internal window touch information management table 176 and the external window touch information management table 178 is similar to the touch information management table 172 in the third embodiment.

[0150] [4.2 Processing flow] The process executed by the internal window engine unit 106 in this embodiment will be described with reference to Fig. 27. First, when a touch event is notified, the internal window engine unit 106 determines whether or not to update touch information (step S400). The process in step S400 is the same as step S304 in Fig. 23.

[0151] If the internal window engine unit 106 does not update the touch information, it determines whether the touched position is inside a transparent region (step S400; No→step S402). If the touched position is not inside a transparent region, the internal window engine unit 106 adds touch information to the internal window (step S402; No→step S404). For example, the internal window engine unit 106 executes the same processes as steps S306 to S310 in FIG. 23 to store the touch ID, coordinates, and operation in touch information for which the touch presence / absence field is "absent" among the touch information stored in the internal window touch information management table 176. On the other hand, if the touched position is inside a transparent region, the internal window engine unit 106 adds touch information to the external window (step S402; Yes→step S406). For example, the internal window engine unit 106 stores the touch ID, coordinates, and action in the touch information stored in the external window touch information management table 178, for which the touch presence / absence is "absent," by processing similar to that in step S404.

[0152] On the other hand, if the touch information is to be updated, the internal window engine unit 106 executes a touch information update process (step S400; Yes→step S408). The touch information update process will be described later.

[0153] Next, the internal window engine unit 106 determines whether the touch information of the external window has been updated (step S410). For example, if touch information has been added or updated in the external window touch information management table 178, the internal window engine unit 106 determines that the touch information of the external window has been updated. If the touch information of the external window has been updated, the internal window engine unit 106 notifies the browser control unit 110 of an operation (touch event) based on the touch information stored in the external window touch information management table 178 as a touch event for the external window (step S410; Yes→step S412). At this time, the internal window engine unit 106 subtracts a value corresponding to the height of the system area from the Y coordinate information and notifies the browser control unit 110. On the other hand, if the touch information of the external window has not been updated, the internal window engine unit 106 omits the processing in step S412 (step S410; No).

[0154] Furthermore, if there is touch information for the internal window, the internal window engine unit 106 processes the touch operation based on the touch information as a touch operation on the internal window (step S414; Yes -> step S144). For example, the internal window engine unit 106 processes a touch operation based on touch information for which the touch presence / absence is "present" among the touch information stored in the internal window touch information management table 176 as a touch operation on the internal window. Note that if there is no touch information for the internal window (if no touch information for which the touch presence / absence is "present" is stored in the internal window touch information management table 176), the internal window engine unit 106 omits the processing in step S144 (step S414; No).

[0155] Next, the flow of the touch information update process will be described with reference to Fig. 28. First, the internal window engine unit 106 identifies touch information to be updated from among the touch information stored in the internal window touch information management table 176 or the external window touch information management table 178 (step S450). Next, the internal window engine unit 106 determines whether the coordinates before update stored in the identified touch information are coordinates within a transparent area (step S452).

[0156] If the coordinates before the update are not within the transparent region, the internal window engine unit 106 determines whether the coordinates after the update are within the transparent region (step S452; No→step S454). If the coordinates after the update are not within the transparent region, the internal window engine unit 106 updates the touch information identified in step S450 based on the touch event notified in step S140 (step S454; No→step S456). In this case, the touch position has not changed from outside the transparent region before and after the update of the touch information, so the touch information of the internal window is updated.

[0157] On the other hand, if the internal window engine unit 106 determines in step S454 that the updated coordinates are coordinates within a transparent region, it clears the touch information identified in step S450 (touch information of the internal window) (step S454; Yes → step S458). Furthermore, the internal window engine unit 106 adds touch information for the external window by processing similar to that of step S406 in Fig. 27 (step S460). As a result, when the touch position of a touch operation on an area other than a transparent region moves to a transparent region, the internal window engine unit 106 can treat the touch operation on the transparent region as an operation on the external window.

[0158] Furthermore, if the internal window engine unit 106 determines in step S452 that the coordinates before the update are within a transparent region, it determines whether the coordinates after the update are within a transparent region (step S452; Yes → step S462). If the coordinates after the update are within a transparent region, the internal window engine unit 106 updates the touch information identified in step S450 based on the touch event notified in step S140 (step S462; Yes → step S464). In this case, the touch position has not changed from within the transparent region before and after the update of the touch information, so the touch information of the external window is updated.

[0159] On the other hand, if the internal window engine unit 106 determines in step S462 that the updated coordinates are not coordinates within a transparent region, it clears the touch information (touch information of the external window) identified in step S450 (step S462; No -> step S466). Furthermore, the internal window engine unit 106 adds touch information for the internal window by processing similar to step S404 in Fig. 27 (step S468). As a result, when the touch position of a touch operation on a transparent region moves to a region other than the transparent region, the internal window engine unit 106 can treat the touch operation on the region other than the transparent region as an operation on the internal window.

[0160] [4.3 Example of operation] An example of operation in this embodiment will be described with reference to Fig. 29 and Fig. 30. Fig. 29 and Fig. 30 are diagrams showing a display screen W400 including an area E400 in which an internal window is displayed and an area E402 in which an external window is displayed (transparent area of ​​the internal window), T400 showing the contents stored in internal window touch information management table 176, and T402 showing the contents stored in external window touch information management table 178. Note that T400 and T402 indicate, from left to right, a touch number, presence or absence of a touch, an X coordinate, a Y coordinate, and an operation, and the numbers included in display screen W400 correspond to the touch numbers of the touch information stored in the touch information management table of the corresponding area.

[0161] 29(a) is a diagram showing a case where no touch operation is performed. When no touch operation is performed, the touch information stored in the internal window touch information management table 176 and the external window touch information management table 178 is cleared.

[0162] 29(b) is a diagram showing a case where a first touch operation is performed on an external window. As shown in T402 of FIG. 29(b), the first touch information is added to the external window touch information management table 178 as touch information with a touch number of 1 (M410).

[0163] 29(c) is a diagram showing a case where a second touch operation is newly performed on the internal window while a first touch operation is being performed. As shown in T400 of FIG. 29(c), the second touch information is added to the internal window touch information management table 176 as touch information with touch number 1 (M420).

[0164] FIG. 30(a) is a diagram showing a case where the touch position of a touch operation managed as touch information with touch number 1 in the external window touch information management table 178 is moved (dragged) to the internal window. When crossing from the external window to the internal window, the touch operation on the external window is deemed to have ended, and the corresponding touch information is cleared from the external window touch information management table 178 (M432) and added to the internal window touch information management table 176 (M430). Note that in FIG. 30(a), since the touch information with touch number 2 has been cleared in the internal window touch information management table 176, the touch information of the touch operation that moved the touch position to the internal window is managed as the second touch information of the internal window. As a result, it is processed as if a second touch operation has started on the internal window.

[0165] 30(b) is a diagram showing a case where the touch operation corresponding to the touch information having the touch number 2 in the internal window touch information management table 176 has ended. In this case, the corresponding touch information is cleared from the internal window touch information management table 176 (M440).

[0166] 30(c) is a diagram showing a case where the touch position of a touch operation managed as touch information with touch number 1 in the internal window touch information management table 176 is moved (dragged) to an external window. In this case, the corresponding touch information is cleared in the internal window touch information management table 176 (M450) and added to the external window touch information management table 178.

[0167] The internal window engine unit 106 processes the touch operation based on the touch information stored in the internal window touch information management table 176. The internal window engine unit 106 also notifies the browser control unit 110 of the touch information stored in the external window touch information management table 178. The touch information is notified from the browser control unit 110 to the external window engine unit 108 via the display control unit 104, and the external window engine unit 108 processes the touch operation based on the notified touch information.

[0168] In this way, when a touch operation is performed across windows, the image forming apparatus of this embodiment can process the touch operation as an operation in the window where the touch position is located.

[0169] [5. Modifications] The present invention is not limited to the above-described embodiments and various modifications are possible. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the present invention are also included in the technical scope of the present invention. For example, the above-described embodiments can be expanded to display two or more windows and to finely control the security layer for each window. In this case, the number of windows can be set to three, and a native GUI can be displayed in the third window.

[0170] Although the above-described embodiments are described separately for convenience, they may be combined within the scope of technical feasibility. For example, the second embodiment and the third embodiment may be combined. In this case, the image forming apparatus can display a native GUI and appropriately process multi-touch operations.

[0171] In addition, the programs that run on each device in the embodiments are programs that control the CPU, etc. (programs that make a computer function) so as to realize the functions of the above-described embodiments. Information handled by these devices is temporarily stored in a temporary storage device (e.g., RAM) during processing, and then stored in various storage devices such as ROMs (Read Only Memories) and HDDs, and is read, modified, and written by the CPU as needed.

[0172] Here, the recording medium for storing the program may be any of semiconductor media (e.g., ROM, non-volatile memory card, etc.), optical recording media / magneto-optical recording media (e.g., DVD (Digital Versatile Disc), MO (Magneto Optical Disc), MD (Mini Disc), CD (Compact Disc), BD (Blu-ray (registered trademark) Disc), etc.), magnetic recording media (e.g., magnetic tape, flexible disk, etc.), etc. Furthermore, not only are the functions of the above-described embodiments realized by executing the loaded program, but the functions of the present invention may also be realized by processing in cooperation with an operating system or other application programs, etc., based on instructions from the program.

[0173] Furthermore, when distributing the program on the market, the program can be stored in a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server computer is of course included in the present invention. [Explanation of symbols]

[0174] 10, 12, 14 Image forming device 100 control section 102 Image processing section 104 Display control unit 106 Internal window engine 108 External Window Engine 110 Browser Control Section 112 HTTP Server 120 Image input unit 130 Image forming unit 132 Paper feed section 134 Printing section 140 Display section 150 Operation section 160 Storage section 162 Operating Systems 164 Web Browser Applications 166 Browser Controller Application 168 Content data storage area 170 Screen setting information storage area 172 Touch Information Management Table 174 Window Information 176 Touch information management table for internal windows 178 Touch information management table for external windows 190 Communications Department 195 Power supply section

Claims

1. A display unit and a control unit are provided, The control unit a first display screen including internal content and a second display screen including external content, the second display screen being positioned behind the first display screen and being superimposed on the first display screen, on the display unit; processing an operation on a transparent area provided on the first display screen as an operation on the second display screen, and processing an operation on an area other than the transparent area as an operation on the first display screen; displaying an input object for inputting to the external content on the first display screen based on an operation on the external content; A result response according to the operation on the input object is notified to the external content. A display device characterized by:

2. The display device described in Claim 1, characterized in that the result response is information including information indicating the selected button and information about the input string.

3. 2. The display device according to claim 1, wherein the input object is a software keyboard or a dialog.

4. The display device described in claim 1 or 2, characterized in that if another touch operation is performed after a touch operation on the transparent area is started and before the touch operation is ended, the control unit processes the other touch operation and a touch operation input in conjunction with the other touch operation as an operation on the second display screen.

5. The display device according to claim 1 or 2, characterized in that, when a touch position on an area other than the transparent area moves into an area of ​​the transparent area, the control unit processes the touch operation on the transparent area as an operation on the second display screen.

6. The display device described in claim 1 or 2, characterized in that the display device operates in an environment in which security restrictions restrict the display of the internal content and the external content on one screen.

7. A step of displaying a first display screen including internal content and a second display screen including external content and positioned behind the first display screen and superimposed on the first display screen; processing an operation on a transparent area provided on the first display screen as an operation on the second display screen, and processing an operation on an area other than the transparent area as an operation on the first display screen; displaying, on the first display screen, an input object for inputting to the external content based on an operation on the external content; notifying the external content of a result response according to the operation on the input object; A control method comprising:

Images