Display device and control method

Abstract

Provided are a display device and the like that can appropriately perform processing of operations in a case where a plurality of screens are displayed in superposition. The display device includes a display section and a control section. The control section displays a first display screen that can include a see-through region on the display section and a second display screen that is located behind the first display screen and is displayed in superposition with the first display screen. The control section processes an operation on the see-through region as an operation on the second display screen and processes an operation on a region other than the see-through region as an operation on the first display screen.

Description

Technical Field

[0001] This invention relates to display devices, etc. Background Technology

[0002] Previously, technologies were proposed that incorporated display units in various devices to improve user convenience.

[0003] For example, an information processing apparatus has been proposed, comprising: a display unit that overlays a rear view image behind a front view image that is displayed through a front view image; a front surface touch panel that receives operations on the front view image; and a rear touch panel that receives operations on the rear view image and is disposed independently of the front surface touch panel (see, for example, Patent Document 1).

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2015-111341 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] The user interface (UI) of information processing devices such as digital multifunction printers (image forming apparatuses) used by multiple users in offices is often a single-screen structure due to the limited functionality and small screen size of these devices. That is, the device does not output screens that are multiplexed using window systems such as personal computers. Even when using a window system, it displays a single full-screen window. Furthermore, in recent years, image forming apparatuses and similar devices have required network access, so web browsers are sometimes installed, and the UI is mounted on these browsers. Web browsers can manage and display multiple pieces of content. Therefore, even with a single-screen structure like the UI of an image forming apparatus, user convenience can be improved by simultaneously displaying / operating both internal device content (internal content) and external content (content obtained from external servers or other external devices).

[0009] Here, when displaying a single screen (a full-screen display of a single window), the iframe tag of HTML (Hypertext Markup Language) is typically used to composite and display the internal and external content. However, due to security constraints such as cross-domain restrictions, there are situations where it is impossible to composite and display the internal and external content within a single screen (a full-screen display of a single window). That is, in the web browser mounted on the image forming apparatus, if it is desired to simultaneously manage and display internal content (system areas such as copy screens and scan screens) and external content (cloud services on the Internet), it is sometimes impossible to composite and display them within a single screen. To solve this problem, it is possible to display the internal and external content in different windows. In this case, there is a need to be able to input the same operations as those performed on the screen structure, but this issue has not been considered in the prior art, such as Patent Document 1.

[0010] In view of the above-mentioned problems, the present disclosure aims to provide a display device or the like that can appropriately process operations when multiple screens are displayed in an overlapping manner.

[0011] Solution for solving the problem

[0012] To solve the above-mentioned problems, the display device disclosed herein includes a display unit and a control unit. The control unit displays a first display screen that includes a transparent area and a second display screen located behind and overlapping the first display screen on the display unit. Operations on the transparent area are processed as operations on the second display screen, and operations on areas other than the transparent area are processed as operations on the first display screen.

[0013] The control method disclosed herein is a control method for a display device, including a display step, which displays a first display screen that includes a transmittance area and a second display screen located behind and overlapping the first display screen; and a processing step, which processes operations on the transmittance area as operations on the second display screen, and processes operations on areas other than the transmittance area as operations on the first display screen.

[0014] Invention Effects

[0015] According to this disclosure, a display device or the like can be provided that can appropriately process operations when multiple screens are displayed overlapping. Attached Figure Description

[0016] Figure 1 This is a perspective view showing the appearance of the image forming apparatus in the first embodiment.

[0017] Figure 2 This is a diagram illustrating the functional configuration of the image forming apparatus in the first embodiment.

[0018] Figure 3 This is a diagram illustrating an example of the data structure of the screen setting information in the first embodiment.

[0019] Figure 4 This is a diagram illustrating an outline of the process in the first embodiment.

[0020] Figure 5 This is a diagram illustrating an outline of the process in the first embodiment.

[0021] Figure 6 This is a diagram illustrating an outline of the process in the first embodiment.

[0022] Figure 7 This is a flowchart illustrating the main processing flow of the image forming apparatus in the first embodiment.

[0023] Figure 8 This is a flowchart illustrating the process executed by the browser control unit in the first embodiment.

[0024] Figure 9 This is a flowchart illustrating the process executed by the display control unit in the first embodiment.

[0025] Figure 10 This is a flowchart illustrating the process executed by the internal window engine unit in the first embodiment.

[0026] Figure 11 This is a diagram illustrating an example of operation in the first embodiment.

[0027] Figure 12 This is a diagram illustrating an example of operation in the first embodiment.

[0028] Figure 13 This is a diagram illustrating an example of operation in the first embodiment.

[0029] Figure 14 This is a diagram illustrating an outline of the process in the second embodiment.

[0030] Figure 15 This is a flowchart illustrating the process executed by the external window engine unit in the second embodiment.

[0031] Figure 16 This is a flowchart illustrating the process executed by the external window engine unit in the second embodiment.

[0032] Figure 17 This is a flowchart illustrating the process executed by the display control unit in the second embodiment.

[0033] Figure 18 This is a flowchart illustrating the process executed by the browser control unit in the second embodiment.

[0034] Figure 19 This is a flowchart illustrating the process executed by the internal window engine unit in the second embodiment.

[0035] Figure 20 This is a diagram illustrating an example of operation in the second embodiment.

[0036] Figure 21 This is a diagram illustrating the functional structure of the image forming apparatus in the third embodiment.

[0037] Figure 22 This is a diagram illustrating an example of the data structure for touch information in the third embodiment.

[0038] Figure 23 This is a flowchart illustrating the processing flow executed by the internal window engine unit in the third embodiment.

[0039] Figure 24 This is a diagram illustrating an example of operation in the third embodiment.

[0040] Figure 25 This is a diagram illustrating an example of operation in the third embodiment.

[0041] Figure 26 This is a diagram illustrating the functional configuration of the image forming apparatus in the fourth embodiment.

[0042] Figure 27 This is a flowchart illustrating the processing flow performed by the internal window engine unit in the fourth embodiment.

[0043] Figure 28 This is a flowchart illustrating the touch information update process in the fourth embodiment.

[0044] Figure 29 This is a diagram illustrating an example of operation in the fourth embodiment.

[0045] Figure 30 This is a diagram illustrating an example of operation in the fourth embodiment. Detailed Implementation

[0046] Hereinafter, an embodiment for carrying out the present disclosure will be described with reference to the accompanying drawings. Furthermore, the following embodiment is merely an example for illustrating the present disclosure, and the technical scope of the invention as described in the claims is not limited to the following description.

[0047] [1. First Implementation Method]

[0048] [1.1 Functional Composition]

[0049] The first embodiment is described with reference to the accompanying drawings. Figure 1 This is a perspective view of the image forming apparatus 10 according to the first embodiment. Figure 2 This is a block diagram showing the functional structure of the image forming apparatus 10.

[0050] Image forming apparatus 10 is an information processing device with functions such as copying, scanning, document printing, and faxing; it is also known as MFP (Multi-Function Printer / Peripheral). For example... Figure 2 As shown, the image forming apparatus 10 includes: a control unit 100, an image input unit 120, an image forming unit 130, a display unit 140, an operation unit 150, a storage unit 160, a communication unit 190, and a power supply unit 195 that supplies power to each functional unit of the image forming apparatus 10.

[0051] The control unit 100 is a functional unit for controlling the entire image forming apparatus 10. The control unit 100 implements various functions by reading and executing various programs stored in the storage unit 160, and may be configured as, for example, one or more arithmetic units (CPU (Central Processing Unit)). Alternatively, the control unit 100 may also be configured as a System-on-a-Chip (SoC) with multiple functions, as described below.

[0052] The control unit 100 functions as the image processing unit 102, display control unit 104, inner window engine unit 106, outer window engine unit 108, browser control unit 110, and HTTP (Hypertext Transfer Protocol) server unit 112 by executing programs stored in the storage unit 160. Here, the display control unit 104, inner window engine unit 106, and outer window engine unit 108 are implemented by executing the web browser application 164, which will be described later. Furthermore, the browser control unit 110 is implemented by executing the browser controller application 166, which will be described later.

[0053] The image processing unit 102 performs various image-related processing. For example, the image processing unit 102 performs sharpening processing and grayscale conversion processing on the image input from the image input unit 120.

[0054] The display control unit 104 displays two windows on the display unit 140: an internal content window (hereinafter referred to as the "internal window") which serves as the first display screen, and an external content window (hereinafter referred to as the "external window") which serves as the second display screen. Furthermore, the display control unit 104 enables the internal window and the external window to process user input on the internal window and the external window.

[0055] The inner and outer windows depict the screen processed by the web browser's display engine (HTML (HyperText Markup Language) rendering engine).

[0056] An external window is a window that displays content (such as cloud services) managed by an external device and located on a network such as the Internet. An internal window is a window (display area) that displays content (internal content) managed and stored within the image forming apparatus 10, and a specified area is allowed to pass through. When displaying external content, the internal window allows the content of the external window to be displayed on the display unit 140 by setting the area displaying the external content as a transparent area.

[0057] The display control unit 104 displays the inner window and the outer window overlapping each other on the display unit 140. The display control unit 104 displays the inner window in the entire display area of ​​the display unit 140, positioned closer to the front (foreground) than the outer window. Conversely, the display control unit 104 displays the outer window superimposed on the inner window, positioned further inward (behind). The front-back relationship (Z-direction) between the inner and outer windows is fixed; the inner window displayed closer to the front and the outer window displayed further inward do not swap positions.

[0058] The display control unit 104 allows a portion of the inner window to be visible based on the displayed screen (content). In this embodiment, the visible area is referred to as the visible area. When the inner window includes the visible area, the display unit 140 displays a screen showing the content of the outer window displayed in the visible area.

[0059] In this embodiment, the internal content includes a system area at the top. The system area is an area containing information about the image forming apparatus 10, buttons for switching functions, etc., and its position and extent (height, etc.) are predetermined. The display control unit 104 displays the system area regardless of whether the internal window includes a transparent area. On the other hand, the external content does not include a system area. Since the external window does not display a system area, its vertical (Y-axis direction) size is smaller than that of the internal window.

[0060] The inner window engine unit 106 interprets the HTML-generated image (content) and displays it in the inner window, or executes a JavaScript program (registered trademark) called from that content. In other words, the inner window engine unit 106 is the engine (HTML rendering engine) for the inner window. Conversely, the outer window engine unit 108 is the engine (HTML rendering engine) for the outer window.

[0061] In this embodiment, the part (engine) that interprets HTML and generates the screen is referred to as the browser engine layer. In this embodiment, the browser engine layer is described in two parts: the internal window engine section 106 for internal windows and the external window engine section 108. However, the browser engine layer may also be an engine shared by both internal and external windows.

[0062] The web browser of this embodiment is implemented through the display control unit 104, the internal window engine unit 106, and the external window engine unit 108 described above.

[0063] The processing performed by the display control unit 104, the internal window engine unit 106, and the external window engine unit 108 will be explained later.

[0064] The browser control unit 110 controls the web browser by processing notification operations and other information. Furthermore, the browser control unit 10 can perform HTTP communication (WebSocket-based communication) to conduct prescribed communication with the internal window engine unit 106. The processing performed by the browser control unit 110 will be described later. In this embodiment, notification includes sending and receiving prescribed information. In this case, the party sending the notification sends information to the party being notified, and the party being notified receives the information.

[0065] HTTP server unit 112 sends HTML (HyperText Markup Language) data, CSS (Cascading Style Sheets) data, and image data according to the HTTP protocol. Upon receiving an HTTP request, HTTP server unit 112 sends the requested data to the source (client) that sent the HTTP request.

[0066] The image input unit 120 inputs image data to the image forming apparatus 10. For example, the image input unit 120 may be composed of a scanning device or the like capable of reading images and generating image data. The scanning device may, for example, use an image sensor such as a CCD (Charge Coupled Device) or a CIS (Contact Image Sensor) to convert the image into an electrical signal, and generate digital data by quantizing and encoding the electrical signal.

[0067] The image forming unit 130 forms (prints) an image on a recording medium such as recording paper. For example, the image forming unit 130 is configured as a laser printer using an electrophotographic method. The image forming unit 130 includes a paper feeding unit 132 and a printing unit 134. The paper feeding unit 132 supplies recording paper. The paper feeding unit 132 is configured as a paper feed tray or a manual paper feed tray, etc. The printing unit 134 forms an image (prints) on the surface of the recording paper and discharges the recording paper from the paper output tray.

[0068] The display unit 140 displays various information. The display unit 140 is composed of display devices such as LCD (Liquid crystal display), organic EL (electro-luminescence) display, and micro LED display.

[0069] The operation unit 150 accepts operations from the user using the image processing device 10. The operation unit 150 is composed of an input device such as a touch sensor. The input to the touch panel can be detected by common detection methods such as resistive film, infrared, electromagnetic induction, or capacitive sensing. Alternatively, a touch panel that integrates the display unit 140 and the operation unit 150 can be mounted on the terminal device 10.

[0070] The storage unit 160 stores various programs and data required for the operation of the image processing device 10. The storage unit 160 may be composed of, for example, a semiconductor memory such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0071] Storage unit 160 stores operating system 162, web browser application 164, and browser controller application 166. Furthermore, storage unit 160 ensures that content data storage area 168 and screen setting information storage area 170 are used as storage areas.

[0072] Operating system 162 is software that serves as the base for operating the image forming apparatus 10. Operating system 162 executes programs by being read and executed by control unit 100, or detects operations input via operation unit 150 and sends information about the detected operations (event information) to the programs. Furthermore, operating system 162 can also provide a platform for executing programs and sending / receiving event information.

[0073] Web browser application 164 is a program that enables control unit 100 to perform the functions of display control unit 104, internal window engine unit 106, and external window engine unit 108. Browser controller application 166 is a program that enables control unit 100 to perform the functions of browser control unit 110.

[0074] The content data storage area 168 stores content data used to display the screen shown in the internal window (content inside the image forming apparatus 10). Content data may include, for example, HTML data, CSS data, and image data.

[0075] The screen setting information storage area 170 stores screen setting information (screen setting information) for the screen displayed on the display unit 140. The screen setting information includes, for example,... Figure 3 As shown, this includes the screen name of the identification screen (e.g., "login screen"), the display settings of the inner window (e.g., "show"), the display settings of the outer window (e.g., "not show"), and the URL representing the destination for retrieving the content (Uniform Resource Locator, e.g., "http: / / localhost / login").

[0076] The display settings for the internal window are stored as either "Display" or "Partially Display". "Display" means displaying the internal window without including the transparent area. "Partially Display" means displaying the internal window including the transparent area. In this embodiment, the transparent area is the area displaying external content, defined as the area outside the system area within the internal content.

[0077] The display settings for the external window can store either "Display" (indicating the external window is displayed) or "Non-Display" (indicating the external window is not displayed). Additionally, in the "Non-Display" setting, the external window can also be displayed as a blank page (about:blank) in standby mode.

[0078] The communication unit 190 communicates with external devices via a LAN (Local Area Network) or WAN (Wide Area Network). The communication unit 190 may consist of, for example, a communication device or module such as a NIC (Network Interface Card) used in a wired / wireless LAN. Furthermore, the communication unit 190 can communicate with another device via a telephone line. In this case, the communication unit 190 is configured with an interface (terminal) that allows insertion of a cable for connection to a telephone line, and performs fax communication and image transmission / reception with another device using known standards or protocols such as G3 / G4 standards.

[0079] [1.2 Processing Overview]

[0080] [1.2.1 Internal and External Windows]

[0081] Reference Figure 4 This section explains the relationship between internal and external windows. Figure 4 (a) of (1) represents the inner window. The inner window includes the area displaying the system area. Figure 4 (a) of (2)) and the area that displays content contained outside the system area ( Figure 4 (a) of (3), hereinafter referred to as the “content display area”).

[0082] on the other hand, Figure 4 (a) of (4) represents the outer window. The outer window is a window of the same size as the content display area. Furthermore, the outer window is positioned at the same location as the content display area of ​​the inner window. Because the inner window ( Figure 4 (a) of (1) must be displayed in a window larger than the outer window. Figure 4 (a) is positioned in front of (4)), so the outer window is hidden by the content display area of ​​the inner window.

[0083] Figure 4 B is a diagram illustrating a display example when internal content (e.g., the operation screen for the copy function or the scan function) is displayed. The settings for the copy and scan functions, and the execution of the job, are displayed in the content display area of ​​the internal window. Figure 4 (b) of (5)).

[0084] Figure 4 (c) is a diagram illustrating an example of display when external content is displayed. In this case, the content display area of ​​the internal window ( Figure 4(c) of (6) becomes a transparent area, displaying the display content (external content) of the external window located inside the content display area of ​​the internal window. As a result, the content within the system area and the external content are displayed in the display unit 140.

[0085] [1.[2.2 Screen Migration]

[0086] Figure 5 This diagram illustrates an example of transitioning from one screen to the next. The image forming apparatus 10 displays the login screen when the power is turned on. Figure 5 (1)) User verification is performed. After user verification, the image forming apparatus 10 displays the main screen ( Figure 5 (2)). The main screen is a screen used to allow the user to select the functions (operations) that enable the image forming apparatus 10 to perform.

[0087] The image forming apparatus 10 displays a setting screen from the main screen based on user operation. Figure 5 (3) ), operation screens for various functions. The operation screen includes the operation screen for the copying function ( Figure 5 (4) ), Print hold function operation screen ( Figure 5 (5) ), operation screen of fax function ( Figure 5 (6) ), operation screen of scanning function ( Figure 5 (7) etc. These screens are operation screens of the functions (native functions) of the image forming apparatus 10, and are internal content. On the other hand, cloud service 1 ( Figure 5 (8) Cloud service 2 Figure 5 (9) is a screen that displays external content provided by an external device. Cloud services can be registered via the settings screen. The screen settings information stored in the screen settings storage area 170 is displayed. Figure 5 The various screens shown.

[0088] [1.2.3 Information flow of the operation]

[0089] Figure 6 It is a diagram showing the notification path of operation information (events) when an operation such as touch is performed. Figure 6 (a) indicates the operating system (hereinafter referred to as "OS"). Figure 6 (b) Indicates an internal window, Figure 6 (c) indicates the external window, Figure 6 (d) indicates that the browser control unit 110, Figure 6 (e) indicates the display control unit 104. Furthermore, the screen (content) displayed in the outer and inner windows is generated by the inner window engine unit 106 and the outer window engine unit 108 of the browser engine layer. Additionally, the explanation will focus on the case where the notified event is a touch event related to a touch operation.

[0090] First, the OS notifies the browser control unit 110 of the touch event. Figure 6 (1). The browser control unit 110 uses inter-process communication to notify the web browser of the notified touch events as touch events for internal windows. Figure 6 (2). The display control unit 104 of the web browser processes the notified touch event as an event for the internal window. Figure 6 (3) At this time, while the inner window is displaying the outer content, it determines whether the notified touch event is an operation on the display part of the outer content (the transparent area of ​​the inner window).

[0091] If the internal window determines that the notified touch event is a touch event on external content, it uses HTTP communication (WebSocket) to notify the browser control unit 110 of the touch event. Figure 6 (4). Browser controller 110 uses inter-process communication to notify the web browser of the notified touch events as touch events for the external window. Figure 6 (5) The display control unit 104 of the web browser processes the notified touch event as an event to the external window. Figure 6 (6).

[0092] In addition, Figure 6 In (3), if it is not determined that the internal window is operating on external content, no action is taken. Figure 6 The processing of (4) resulted in no action being taken. Figure 6 The processing of (5) and (6) only treats touch events as events to the inner window.

[0093] Additionally, from the internal window ( Figure 6 (b)), external window ( Figure 6 (c) and display control unit 104 Figure 6 (e) enables web browsing. The web browser communicates with an internal HTTP server (HTTP server unit 112) and an external server, i.e., an external HTTP server on the Internet, to obtain content. Furthermore, the web browser, through the processing of the display control unit 104, displays the obtained content in an internal or external window.

[0094] [1.3 Processing Flow]

[0095] Next, refer to Figures 7 to 10 The process flow performed by the image forming apparatus 10 will be explained. Furthermore, Figures 7 to 10 The processing shown is executed by the control unit 100 reading the program stored in the storage unit 160.

[0096] Here, the control unit 100 reads and executes the operating system 162, thereby causing the OS to operate. Therefore, the control unit 100 detects user input operations (e.g., touch operations input via the operation unit 150). Furthermore, the control unit 100 enables the display control unit 104, the internal window engine unit 106, the external window engine unit 108, the browser control unit 110, and the HTTP server unit 112 to function within the OS. When the OS operated by the control unit 100 detects a user input operation, it notifies the browser controller 110 of the operation (an event) and provides information indicating the content of the operation.

[0097] [1.3.1 Main Processing]

[0098] Reference Figure 7 The flow of the main processing (main processing) performed by the image forming apparatus 10 in this embodiment will be described. Figure 7 The processing shown is performed when updating the screen displayed on the display unit 140.

[0099] First, the control unit 100 reads the screen setting information of the screen displayed on the display unit 140 from the screen setting information storage area 170 based on the user's operation and the state of the image forming apparatus 10 (step S100).

[0100] Next, the control unit 100 applies the display settings of the internal window included in the screen setting information read in step S100 to the internal window (step S102). Furthermore, the control unit 100 applies the display settings of the external window included in the screen setting information read in step S100 to the external window (step S104).

[0101] Next, the control unit 100 displays content (step S106). For example, when the URL included in the screen setting information read in step S100 contains the domain name of the HTTP server unit 112 (e.g., localhost), the control unit 100 causes the internal window to display the content determined by that URL. Furthermore, when the URL included in the screen setting information read in step S100 contains a domain name other than the domain name of the HTTP server unit 112, the control unit 100 causes the external window to display the content determined by that URL.

[0102] [1.3.2 Processing by the Browser Control Unit]

[0103] Reference Figure 8 The processing performed by the browser control unit 110 will be explained. Furthermore, the browser control unit 110 repeatedly executes... Figure 8 The processing shown.

[0104] First, the browser controller 110 determines whether a touch event has been notified from the OS (step S120). The touch event is notified along with information indicating the content of the operation (operation information), such as the touched location and the status of the touch operation. The information regarding the status of the touch operation includes information related to the touch operation action, such as setting a new touch position (starting a touch operation), moving the touch position, or removing the touch position (ending the touch operation).

[0105] When a touch event is notified from the OS, the browser control unit 110 uses inter-process communication to notify the browser (display control unit 104) of the touch event as a touch event for an internal window (step S120; yes → step S122).

[0106] On the other hand, if the browser control unit 110 does not notify the touch event from the OS, it determines whether a touch event for the external window has been notified from the internal window (step S120; no → step S124). Furthermore, in this embodiment, the touch event for the external window is notified to the browser control unit 110 from the internal window engine unit 106 using HTTP communication (WebSocket). If the browser control unit 110 is notified of a touch event for the external window, it notifies the browser (display control unit (104)) of the touch event for the external window using inter-process communication (step S124; yes → step S126). Thus, the browser control unit 110 notifies the browser (display control unit 104) of the touch event notified in step S122 as a touch event for the external window. Furthermore, if the browser control unit 110 does not notify the external window of a touch event, it omits the processing in step S126 (step S124; no).

[0107] [1.3.3 Display Control Unit]

[0108] Reference Figure 9 The processing performed by the display control unit 104 will be explained. Furthermore, the display control unit 104 repeatedly performs... Figure 9 The processing shown.

[0109] First, the display control unit 104 determines whether a touch event for the internal window has been notified from the browser control unit 110 (step S130). If a touch event for the internal window has been notified, the display control unit 104 processes the touch event as a touch event for the internal window (step S130; if yes → step S132). For example, the display control unit 104 notifies the internal window engine unit 106 (browser engine layer) of the touch event.

[0110] On the other hand, if no touch event is notified for the internal window, the display control unit 104 determines whether a touch event for the external window has been notified from the browser control unit 110 (step S130; No → step S134). If a touch event for the external window has been notified, the display control unit 104 processes the touch event as a touch event for the external window (step S134; Yes → step S136). For example, the display control unit 104 notifies the external window engine unit 108 (browser engine layer) of the touch event. Furthermore, if no touch event for the external window has been notified, the browser control unit 110 omits the processing in step S136 (step S134; No).

[0111] [1.3.4 Internal Window Engine Section]

[0112] Reference Figure 10 The processing performed by the internal window engine unit 106 will be explained. Furthermore, the internal window engine unit 106 repeatedly executes... Figure 10 The processing shown.

[0113] First, the internal window engine unit 106 determines whether a touch event has been notified from the display control unit 104 (step S140). If the internal window engine unit 106 determines that no touch event has been notified, it repeats the process of step S140 (step S140; no).

[0114] On the other hand, when the internal window engine unit 106 is notified of a touch event, it determines whether a touch operation targeting the through area has been performed based on the operation information notified along with the touch event (step S140; yes → step S142). If the internal window engine unit 106 has not performed a touch operation targeting the through area, it processes the touch operation as a touch operation targeting the internal window (step S142; no → step S144). On the other hand, if it has performed a touch operation targeting the through area, the internal window engine unit 106 uses HTTP communication (WebSocket) to notify the browser control unit 110 of the touch event notified in step S140 as a touch event for the external window (step S142; yes → step S146).

[0115] [1.3.5 External Window Engine Section]

[0116] When the external window engine unit 108 receives a touch event notification from the display control unit 104 for an external window, it processes the touch operation based on the touch event.

[0117] Thus, by execution Figures 7 to 10The processing shown involves notifying the outer window engine unit 108 of operations performed through the through area as touch events for the outer window, and processing them as operations on the outer window. Additionally, the inner window engine unit 106 processes operations on areas outside the through area as operations on the inner window.

[0118] [1.4 Action Examples]

[0119] An example of the operation in this embodiment will be explained. Figure 11 (a) is an example of a login screen W100. The login screen W100 is displayed when user authentication is set to be valid in the setting screen of the image forming apparatus 10. The login screen W100 includes a system area E100 and a content display area E102. The content of the login screen is displayed in the content display area E102. The content of the login screen includes a username input field T100, a password input field T102, and a login button B100, etc.

[0120] Figure 11 (b) is an example of the display screen W110 when the username input field T100 or the password input field T102 is touched. When the input field is touched, the image forming apparatus 10 of this embodiment displays a soft keyboard in an internal window. Soft keyboard functionality is generally provided through an OS or platform, but in the case of embedded devices such as the image forming apparatus 10, it is sometimes not provided. In this case, the image forming apparatus 10 implements the soft keyboard functionality using HTML and JavaScript in an internal window.

[0121] Figure 11 (c) is an example of displaying screen W120 when dialog box E120 is displayed. Dialog box E120 is displayed to notify the user of specified information, such as when the password entered in registration screen W100 is incorrect. When the function of displaying dialog box is not provided by the OS or platform, the image forming apparatus 10 implements the function of displaying dialog box using HTML and JavaScript in an internal window.

[0122] In this specification, the soft keyboard and dialog boxes are referred to as the native GUI (Graphical User Interface). The native GUI is a component (GUI, UI) that serves as an input object for performing prescribed input operations, such as button selection and string input. The image forming apparatus 10 implements the input function by displaying a component (input object) with the same functionality as the native GUI through an internal window. Furthermore, in the following description, components (input objects) that perform the same functions as the native GUI displayed in the internal window will only be described as the native GUI.

[0123] Figure 12 (a) is an example of the main screen W130, which is the initial screen displayed when login is successful or when user authentication is set to invalid in the settings screen. The main screen includes, for example, an area E130 containing function buttons. Function buttons are buttons used to select functions that the image forming apparatus 10 performs. Area E130 includes, for example, four function buttons: copy function button B130, print hold function button B131, fax function button B132, and scan function button B133. In addition, the main screen W130 includes a button B134 for displaying the settings screen, a button B135 for adjusting the volume, and a button B136 for adjusting the brightness of the display unit 140, etc.

[0124] The main screen W130 can be configured by changing the settings or adding function buttons. If not all function buttons can be displayed on one screen, the area E130 of the main screen W130 will scroll left and right depending on the operation of the selection triangle button (button B137 and button B138) or the tap / scroll operation.

[0125] Figure 12 (b) is an example of the display screen W140 when button B136 on the main screen W130 is selected. Display screen W140 includes a pop-up window E140 for adjusting brightness. Additionally, Figure 12 (c) is an example of the display screen W150 when button B135 on the main screen W130 is selected. Display screen W150 includes a pop-up window E150 for adjusting the volume. Pop-up windows E140 and E150 are implemented using HTML and JavaScript.

[0126] Figure 13 (a) is a choice Figure 12(a) shows an example of the main screen W160 when the area E130 of the main screen W130 is scrolled to the right, with button B138 serving as the right-direction triangle button on the main screen W130. The main screen W160 includes function buttons B160 for displaying the content of cloud service 1 and function button B161 for displaying the content of cloud service 2. Users can utilize cloud services (external content) by selecting either function button B160 or function button B161.

[0127] Figure 13 (b) is from Figure 12 (a) shows an example of the operation screen W170 displayed when the copy function button B130 is selected on the main screen W130. The copy function is a native function of the image forming apparatus 10. In addition, the operation screen W170 for the copy function is internal content and is displayed in an internal window.

[0128] Figure 13 (c) is from Figure 13 (a) shows an example of the operation screen W180 displayed when the function button B160 for cloud service 1 is selected on the main screen W160. This example shows the verification screen for cloud service 1. Here, the system area E180, which is the upper area of ​​the internal window, is not visible. The system area includes, for example, the main button B180 for switching the operation screen W180 to the main screen. The content area E181, which is the lower part of the internal window, is visible. Therefore, the screen of the external cloud service is displayed in the content area E181.

[0129] Users can perform touch operations on the operation screen W180. Here, if the user touches an area outside the visible area of ​​an internal window (system area E180), the operation is processed as a touch operation on the internal window. Therefore, if the user touches the main button B180 within the system area E180, the image forming apparatus 10 switches the operation screen W180 to the main screen as a touch on the main button B180. On the other hand, if the user touches the visible content area E181 (visible area), the operation is processed by the image forming apparatus 10 as a touch operation on the external window.

[0130] Furthermore, while the above embodiments describe the processing of touch operations (touch events), mouse operations (mouse events) can also be notified in the same way.

[0131] Thus, the image forming apparatus of this embodiment is composed of two windows, an inner window and an outer window, and can also be operated by the user using a single screen for touch operation.

[0132] Here, the image forming apparatus of this embodiment displays external content in an external window that is different from the internal window displaying internal content. Therefore, the image forming apparatus of this embodiment can handle situations where, due to cross-domain constraints, it is impossible to use an iframe tag to display both internal and external content.

[0133] Typically, to avoid cross-domain restrictions, cross-domain settings need to be allowed on the external HTTP server side. However, this increases the management burden on the external content side (external HTTP server side) and makes it difficult to manage when the settings on the external HTTP server side (cloud service side) cannot be changed. In particular, the external HTTP server sometimes implements cross-domain restrictions to handle click detection when displaying content using iframe tags, which can lead to reduced security due to setting changes. To address these issues, the image forming apparatus of this embodiment does not require changing the settings on the external HTTP server side. Instead, it sets the UI on the image forming apparatus side (client side) to a dual-window configuration, thereby solving the problem. Furthermore, the image forming apparatus of this embodiment has a dual-window structure, allowing users to perform touch operations on a single screen simultaneously, improving user convenience. The image forming apparatus of this embodiment has a dual-window structure, eliminating the need for touch operations that switch between windows, enabling seamless touch operations without compromising the usability of a single-window structure.

[0134] [2. Second Implementation]

[0135] Next, the second embodiment will be described. The second embodiment is an implementation that, in addition to the processing described in the first embodiment, performs processing of the native GUI of the external window based on the operation of the external window.

[0136] In the first embodiment, displaying the native GUI in an inner window was described. However, there are cases where the native GUI cannot be displayed in an outer window (a single window). This is due to limitations such as iframes; for example, in a web browser displaying external content (the outer window), the soft keyboard, which is internal content, cannot be displayed. Thus, the native GUI, which would otherwise be displayed in the same window, sometimes cannot be displayed in the same window.

[0137] Therefore, the image forming apparatus 10 of this embodiment implements a native GUI in a dedicated window (internal window) that ensures security, allowing external windows to utilize this native GUI to implement the browser's native GUI while ensuring security. Thus, the image forming apparatus 10 enables users to input external content and reflects the user's input onto the external content.

[0138] In this embodiment, the native GUI implemented by the internal window is shown below.

[0139] (1) Soft keyboard

[0140] A soft keyboard is a software-implemented keyboard that displays the keys normally located on a physical keyboard, the [OK] button, and the [Cancel] button. When the user selects the [OK] button, the input (string) entered using the keys is reflected in the content displayed in the internal and external windows.

[0141] (2) Dialog Box

[0142] A dialog box is a window (dialog box) displayed to show information or to request the user to select a button or input information. In this embodiment, the following four types of dialog boxes are displayed.

[0143] (2-1) JavaScript alert dialog box

[0144] A JavaScript alert dialog box is a dialog box that contains a message and an [OK] button. In a JavaScript program, the alert dialog box is displayed by performing processing to show it.

[0145] (2-2) JavaScript's confirm dialog box

[0146] A JavaScript confirm dialog box is a dialog box that includes a message, an [OK] button, and a [Cancel] button. In a JavaScript program, the confirm dialog box is displayed by executing the necessary processing.

[0147] (2-3) JavaScript prompt dialog box

[0148] A JavaScript prompt dialog box is a dialog box that includes a message, a string input field, an [OK] button, and a [Cancel] button. In a JavaScript program, the prompt dialog box is displayed by executing the necessary code.

[0149] (2-4) Verification dialog box

[0150] The verification dialog box is displayed when the server returns an HTTP 401 (verification failed, the HTTP response is 401) error. The verification dialog box includes a string input field for entering the username and a string input field for entering the password, two input fields for entering verification information, and also includes [OK] and [Cancel] buttons.

[0151] In addition, in this embodiment, the JavaScript alert dialog box, the JavaScript confirm dialog box, and the JavaScript prompt dialog box are referred to as JavaScript dialog boxes.

[0152] [2.1 Information Flow]

[0153] Figure 14 This diagram illustrates the notification path of events (information) within the image forming apparatus 10 when an operation occurs that invokes the native GUI from an external window (such as displaying the soft keyboard or displaying a dialog box). Additionally, Figure 14 (a) to (e) are with Figure 6 The same functional parts are (a) to (e).

[0154] First, the web browser (external window) detects an operation or process that displays the native GUI. At this time, the external window engine unit 108 notifies the display control unit 104 of a request to display the native GUI (native GUI startup request). Figure 14 (1)). The display control unit 104 notifies the browser control unit 110 of the native GUI startup request ( Figure 14 (2)). The browser control unit 110 uses HTTP communication (WebSocket) to notify the internal window engine unit 106 of the native GUI startup request ( Figure 14 (3)). The internal window engine unit 106, which was notified of the native GUI startup request, displays the native GUI in the internal window.

[0155] After the user completes the operation on the native GUI, the internal window engine 106 uses HTTP communication (WebSocket) to notify the browser control 110 that the operation on the native GUI has ended (the result of the native GUI operation). Figure 14 (4)). Browser controller 110 uses inter-process communication to notify the browser (display control unit 104) that the operation of the native GUI has ended (operation result). Figure 14 (5)). The web browser (display control unit 104) reflects the operation results in the external content. Figure 14 (6)).

[0156] [2.2 Processing Flow]

[0157] Next, refer to Figures 15 to 19 The process flow of the image forming apparatus 10 will be explained. Figures 15 to 19 The processing shown is executed by the control unit 100 reading the program stored in the storage unit 160. Furthermore, Figures 15 to 19 The processing shown is the same as that in the first embodiment. Figures 7 to 10 The processing shown is performed in parallel.

[0158] [2.2.1 External Window Engine Section (Judgment and Processing)]

[0159] Reference Figure 15 The judgment and processing performed by the external window engine unit 108 will be explained. This judgment and processing determines whether an operation to display the native GUI exists. Furthermore, the external window engine unit 108 repeatedly executes... Figure 15 The processing shown.

[0160] First, the external window engine unit 108 determines whether the verification has failed when the page loads (when content is retrieved) (step S200). For example, if the external window engine unit 108 receives an HTTP response with an HTTP response code of 401 from an external HTTP server, it determines that the verification has failed. In the case of verification failure, the external window engine unit 108 notifies the display control unit 104 of a native GUI startup request for the verification dialog box (step S200: Yes → step S202).

[0161] On the other hand, if the verification during page loading does not fail, the external window engine unit 108 determines whether there is a native GUI startup request for a JavaScript dialog box (step S200; no → step S204). A native GUI startup request for a JavaScript dialog box is a request to display dialog boxes by executing processing in a JavaScript program to display alert dialog boxes, confirm dialog boxes, and prompt dialog boxes. If a native GUI startup request for a JavaScript dialog box exists, the external window engine unit 108 notifies the display control unit 104 of the native GUI startup request for the JavaScript dialog box (step S204; yes → step S206).

[0162] On the other hand, if there is no native GUI startup request for a JavaScript dialog box, the external window engine unit 108 determines whether a text input operation has been performed (step S204; no → step S208). For example, if the external window engine unit 108 performs a touch operation on a string input field displayed via an input label or a textarea label, it determines that a text input operation has been performed. If a text input operation has occurred, the external window engine unit 108 notifies the display control unit 104 of a native GUI startup request for the soft keyboard (step S208; yes → step S210). Furthermore, if there is no text input operation, the external window engine unit 108 omits the processing in step S210 (step S208; no).

[0163] [2.2.2 External Window Engine Section (Result Reflection Processing)]

[0164] Reference Figure 16 The result reflection processing performed by the external window engine unit 108 will be explained. Result reflection processing is the process of reflecting the result response (operation result) of the native GUI to the external window. Furthermore, the external window engine unit 108 repeatedly performs... Figure 16 The processing shown.

[0165] First, the external window engine unit 108 determines whether a result response has been received from the native GUI of the verification dialog box (step S220). The result response from the native GUI of the verification dialog box may include, for example, the username and password entered via the verification dialog box. If a result response from the native GUI of the verification dialog box has been received, the external window engine unit 108 notifies the external HTTP server of the result (the entered username and password) (step S220; Yes → step S222). Furthermore, if the verification performed by the external HTTP server is successful, the display control unit 104 or the external window engine unit 108 continues processing to retrieve content from the external HTTP server or display the retrieved content.

[0166] On the other hand, if the external window engine unit 108 is notified of the result response to the native GUI of the verification dialog box, it determines whether it has been notified of the result response to the native GUI of the JavaScript dialog box (step S220; No → step S224). The result response of the JavaScript dialog box to the native GUI is, for example, information indicating the selected button or information containing the entered string. If the external window engine unit 108 is notified of the result response to the native GUI of the JavaScript dialog box, it reflects the button selected by the user or the entered string in the external content (step S224; Yes → step S226).

[0167] On the other hand, if the external window engine unit 108 is notified of the result response to the native GUI for the JavaScript dialog box, it determines whether it has been notified of the result response to the native GUI for the soft keyboard (step S224; no → step S228). The result response of the soft keyboard to the native GUI is, for example, information including a string entered by the user. If the external window engine unit 108 is notified of the result response to the native GUI for the soft keyboard, it reflects the string entered by the user in the dialog box. Figure 15 In step S208, the selected string input field is selected (step S228; Yes → step S230). Additionally, the external window engine unit 108 omits the processing in step S230 (step S228; No) if no result response is notified to the native GUI for the soft keyboard.

[0168] [2.3 Display Control Unit]

[0169] Reference Figure 17 The processing performed by the display control unit 104 will be explained. Furthermore, the display control unit 104 repeatedly performs... Figure 17 The processing shown.

[0170] First, the display control unit 104 determines whether a native GUI startup request has been notified from the external window engine unit 108 (step S250). If a native GUI startup request has been notified, the display control unit 104 uses inter-process communication to notify the browser control unit 110 of the native GUI startup request (step S250; yes → step S252).

[0171] On the other hand, if no native GUI startup request is requested, the display control unit 104 determines whether a result response has been notified from the browser control unit 110 to the native GUI (step S250; No → step S254). If a result response has been notified, the display control unit 104 notifies the external window engine unit 108 of the notified result response (step S254; Yes → step S256). Alternatively, if no result response has been notified to the native GUI, the display control unit 104 omits the processing in step S256 (step S254; "No").

[0172] [2.2.4 Processing by the Browser Control Unit]

[0173] Reference Figure 18 The processing performed by the browser control unit 110 will be explained. Furthermore, the browser control unit 110 repeatedly executes... Figure 18 The processing shown.

[0174] First, the browser control unit 110 determines whether a native GUI startup request has been notified from the display control unit 104 (step S260). If a native GUI startup request has been notified, the browser control unit 110 uses HTTP communication (WebSocket) to notify the internal window engine unit (106) of the native GUI startup request (step S260; yes → step S262).

[0175] On the other hand, if the browser control unit 110 does not notify the native GUI of the start request, it determines whether the result response has been notified to the native GUI from the internal window engine unit 106 (step S260; no → step S264). If the browser control unit 110 has notified the native GUI of the result response, it uses inter-process communication to notify the web browser (display control unit (104) of the notified result response) (step S264; yes → step S266). Alternatively, if the browser control unit 110 has not notified the native GUI of the result response, it omits the processing in step S266 (step S264; no).

[0176] [2.2.5 Internal Window Engine Section]

[0177] Reference Figure 19 The processing performed by the internal window engine unit 106 will be explained. Furthermore, the internal window engine unit 106 repeatedly executes... Figure 19 The processing is shown.

[0178] First, the inner window engine unit 106 determines whether a native GUI launch request for the verification dialog box has been received from the browser control unit 110 (step S280). If a native GUI launch request for the verification dialog box has been received, the inner window engine unit 106 displays the verification dialog box in the inner window (step S280; if yes → step S282). At this time, the inner window engine unit 106 sets the system area and the area other than the area where the verification dialog box is displayed as transparent areas. As a result, the verification dialog box is overlaid on the outer content.

[0179] When the operation on the verification dialog box is completed, the internal window engine unit 106 uses HTTP communication (WebSocket) to notify the browser control unit 110 of the result response (step S284). For example, if the user selects the [OK] button, the internal window engine unit 106 notifies the browser control unit 110 of the result response containing the username and password entered by the user. Additionally, if the user selects the [Cancel] button, the internal window engine unit 106 notifies the browser control unit 110 of the result response containing information indicating that the [Cancel] button has been selected.

[0180] On the other hand, if the internal window engine unit 106 does not notify the browser control unit 110 of the native GUI launch request for the JavaScript dialog box (step S280; no → step S286), it determines whether the browser control unit 110 has notified the browser control unit 110 of the native GUI launch request for the JavaScript dialog box (step S286; yes → step S288). If the browser control unit 106 has been notified of the native GUI launch request for the JavaScript dialog box, it displays the requested type of JavaScript dialog box in the internal window (step S286; yes → step S288). At this time, the internal window engine unit 106 uses the area other than the system area and the area where the JavaScript dialog box is displayed as the transparent area.

[0181] When the operation on the JavaScript dialog box ends, the internal window engine unit 106 uses HTTP communication (WebSocket) to notify the browser control unit 110 of the result response (step S290). For example, the internal window engine unit 106 notifies the browser control unit 110 of the result response, which includes information about the button selected by the user and information about the string entered by the user.

[0182] On the other hand, if there is no native GUI launch request from the JavaScript dialog box, the inner window engine unit 106 determines whether a native GUI launch request for the soft keyboard has been received from the browser control unit 110 (step S286; no → step S292). If a native GUI launch request for the soft keyboard has been received, the inner window engine unit 106 displays the soft keyboard in the inner window (step S292; yes → step S294). At this time, the inner window engine unit 106 sets the area other than the system area and the area where the soft keyboard is displayed as the transparent area.

[0183] When the operation on the soft keyboard ends, the internal window engine unit 106 notifies the browser control unit 110 of the result response using HTTP communication (WebSocket) (step S296). For example, if the user selects the [OK] button, the internal window engine unit 106 notifies the browser control unit 110 of the result response containing the string entered by the user and information indicating that the [OK] button has been selected. Additionally, if the user selects the [Cancel] button, the internal window engine unit 106 notifies the browser control unit 110 of the result response containing information indicating that the [Cancel] button has been selected. Furthermore, if the internal window engine unit 106 does not notify the native GUI startup request of the soft keyboard, it omits the processing in steps S294 and S296 (step S292; No).

[0184] [2.3 Action Examples]

[0185] Reference Figure 20 Here is an example of the action in this implementation. Figure 20 (a) is an example of a W200 screen displaying the E200 soft keyboard in an internal window. As a native GUI, the E200 soft keyboard is displayed in an internal window when the user touches the string input fields (such as username (ID) or password) from cloud service content displayed in the external window. The internal window displays the upper system area and the soft keyboard, while the area outside is set as a transparent area. Thus, the soft keyboard is displayed overlaid on the external content.

[0186] Figure 20 (b) is an example of the display screen W210 of the JavaScript dialog box E210 in the internal window. For example, the JavaScript dialog box E210 is displayed when the user enters an incorrect password. The JavaScript dialog box E210 is displayed in the internal window in the same way as the soft keyboard. Figure 20 (b) is an example of an alert dialog box that displays a message such as "Incorrect password."

[0187] Furthermore, in the above embodiments, it was described that the native GUI is a soft keyboard and a dialog box. However, the native GUI does not necessarily have to be a soft keyboard and a dialog box, as long as it allows the user to input external content. For example, the image forming apparatus 10 can display a screen for selecting a date and time as the native GUI, or it can display a screen for entering an email address or URL (Uniform Resource Locator).

[0188] Thus, even without a native GUI provided by the operating system, the image forming apparatus of this embodiment can appropriately display the native GUI or reflect operations on the native GUI.

[0189] [3. Third Implementation Method]

[0190] Next, the third embodiment will be described. The third embodiment is an embodiment in which the browser engine layer (internal window engine unit) performs multi-touch operation management processing, in addition to the processing described in the first embodiment. This embodiment is a modification of the first embodiment. Figure 2 Replace with Figure 21 The first embodiment Figure 10 Replace with Figure 23 The figure is shown. Furthermore, the same reference numerals are used to label the same functional parts and processes, and descriptions are omitted.

[0191] In this embodiment, in the two window structures of the inner window and the outer window, the first point of touch is initiated. If the first point of touch or multiple touches are performed, all touch operations are treated as a series of touch operations and processed as touch operations in the window where the first point of touch was performed before all touch operations are completed.

[0192] In this embodiment, when a touch operation is performed across windows—that is, after a first touch, a different window is touched than the one from which the first touch was performed—it is processed as if the window that initiated the initial touch were the window. In other words, while multiple touch operations are being processed as a series of touch operations, the subsequent touch operation is necessarily processed as a touch operation on either the inner or outer window.

[0193] [3.1 Functional Composition]

[0194] Reference Figure 21 This embodiment explains the functional configuration of the image forming apparatus 12. The image forming apparatus 12 and... Figure 2 Compared to the image forming apparatus 10 shown, the storage unit 160 also stores a touch information management table 172 and window information 174.

[0195] Touch Information Management Table 172 is a table used to manage (store) information about touch operations. For example, Touch Information Management Table 172... Figure 22 As shown, the touch number (e.g., "1") that identifies the touch information, the inherent number of the point that contacts the touch surface (operation unit 150) that is the touch ID (e.g., "1"), the presence or absence of touch (e.g., "yes"), the X coordinate (e.g., "600.0") and Y coordinate (e.g., "200.0") that represent the coordinates of the touch are established and stored in correspondence with the touch action (e.g., "start").

[0196] The touch ID is, for example, the touch ID obtained from the event handling of a JavaScript touch operation. The coordinates refer to the coordinates represented as (x, y), where the top-left pixel of the display unit 140 is taken as the origin (0, 0), the number of horizontal pixels from the origin to the pixel of interest is set to x, and the number of vertical pixels is set to y. For example, the touch information management table 172 stores values ​​from 0 to 639 in the X coordinate and values ​​from 0 to 479 in the Y coordinate. The action stores a value of any one of "start", "move", or "end". "start" indicates that a new touch position has been set (touch operation has started). "move" indicates that the touch position has been moved. "end" indicates that the touch position has been removed (touch operation has ended). Furthermore, the initial value of the action is "end".

[0197] Furthermore, in this embodiment, assuming the operation unit 150 is a touch panel capable of 5-point touch, if 6 or more touches are performed, touch events beyond the 6th touch will not be notified. Therefore, touch operation information is managed up to 5 points, and the touch number becomes any value from 1 to 5.

[0198] Window information 174 indicates the window that initiated the first touch. Window information 174 is initially set to NULL, and upon the first touch, it stores information from either the "inner window" or the "outer window." Furthermore, NULL is stored in window information 174 when all touch operations have ended.

[0199] [3.2 Processing Flow]

[0200] Reference Figure 23 The processing performed by the internal window engine unit 106 in this embodiment will be described. First, when the internal window engine unit 106 is notified that a window has been touched, it determines whether the window information 174 is NULL (step S300). If the window information 174 is NULL, the internal window engine unit 106 sets information indicating the touched window in the window information 174 (step S300; Yes → step S302). For example, if the internal window engine unit 106 touches the transparent portion of the internal window, it stores "external window" in the window information 174; otherwise, it stores "internal window" in the window information 174. Furthermore, if the window information 174 is not NULL, the internal window engine unit 106 omits the processing in step S302 (step S300; No).

[0201] Next, the internal window engine unit 106 determines whether to update the touch information managed in the touch information management table 172 (step S304). If the touch operation is an operation corresponding to "move" or "end", the internal window engine unit 106 determines to update the touch information. On the other hand, if the touch operation is an operation corresponding to "start", it determines not to update the touch information (add touch information).

[0202] Without updating the touch information, the internal window engine unit 106 changes the variable n relative to the touch number from 1 to the maximum value of the touch number (5 in this embodiment) (step S306). The internal window engine unit 106 refers to the touch information management table 172 and determines whether the presence or absence of touch stored in the touch information with touch number n is "none" (step S308). If the presence or absence of touch is "none", the internal window engine unit 106 stores the touch ID, coordinates, and action based on the touch event notified in step S140 in the touch information with touch number n, and sets the presence or absence of touch to "yes". Therefore, the internal window engine unit 106 adds touch information to the touch information management table 172 (step S310).

[0203] On the other hand, when updating touch information (step S304; Yes), the internal window engine unit 106 obtains the touch ID based on the touch event notified in step S140. Then, the internal window engine unit 106 updates the touch information storing that touch ID (updating the touch information of the object) based on the touch event notified in step S140 (step S312). Here, if the touch operation is an operation corresponding to "end", the internal window engine unit 106 stores "0.0" in the X and Y coordinates of the touch information of the object to be updated, and initializes (clears) the touch information by setting the presence or absence of touch to "none".

[0204] Next, the internal window engine unit 106 determines whether an "external window" is stored in the window information 174 (step S314). If no "external window" is stored in the window information 174, the internal window engine unit 106 processes the operation based on the touch information stored in the touch information management table 172 as a touch operation for the internal window (step S314; No → step S144). On the other hand, if the window stored in the window information 174 is an "external window", the internal window engine unit 106 notifies the browser control unit 110 of the operation (touch event) based on the touch information stored in the touch information management table 172 as a touch event for the external window (step S314; Yes → step S316). At this time, the internal window engine unit 106 notifies the browser control unit 110 after subtracting the value corresponding to the height of the system area from the information relative to the Y coordinate.

[0205] Next, the internal window engine unit 106 determines whether all the touch information actions stored in the touch information management table 172 are "end" (step S318). If all the touch information actions are "end", the internal window engine unit 106 sets NULL in the window information 174 (step S318; Yes → step S320). Furthermore, if not all the touch information actions are "end", the internal window engine unit 106 omits the processing in step S320 (step S318; No).

[0206] In this way, the internal window engine unit 106 treats other touch operations input after the first touch operation begins and until the touch operation ends, as well as touch operations input in conjunction with those other touch operations, as touch operations of the window that performed the first touch operation. As a result, the internal window engine unit 106 can process a series of touch operations as operations in the window corresponding to the touch position of the first point.

[0207] For example, after a touch operation on the visible area (outer window) begins, other touch operations may sometimes occur before the current touch operation ends. In this case, the inner window engine unit 106 notifies the display control unit 104 of the information (touch events) of these other touch operations and touch operations entered before all other touch operations have ended (touch operations entered in conjunction with other touch operations). Therefore, when other touch operations occur after a touch operation on the visible area (outer window) has begun, the inner window engine unit 106 can process touch operations entered before all touch operations have ended as operations on the outer window. Similarly, when a touch operation on an area other than the visible area (inner window) has begun, the inner window engine unit 106 will process touch operations entered before all touch operations have ended as touch operations on the inner window.

[0208] [3.3 Action Examples]

[0209] Reference Figure 24 and Figure 25 Explain the action example in this implementation method. Figure 24 as well as Figure 25 This diagram represents a display screen W300 containing an area E300 displaying an inner window, an area E302 displaying an outer window (the area through which the inner window is visible), a display T300 displaying the content stored in the touch information management table 172, and a display D300 displaying the content stored in the window information 174. Furthermore, T300, from left to right, indicates the touch number, touch presence / absence, X coordinate, Y coordinate, and action. The numbers displayed on the display screen W300 correspond to the touch numbers.

[0210] Figure 24 (a) is a diagram showing the case where no touch operation is performed. In the case of no touch operation, the touch information stored in the touch information management table 172 is cleared, and NULL is stored in the window information 174.

[0211] Figure 24 (b) is a diagram illustrating the case where the first touch operation is performed on the outer window. For example... Figure 24 (b) As shown in T300, the touch information of the first point (M310) is added to the touch information management table 172. Additionally, as... Figure 24 (b) As shown in D300, the “external window” is stored in window information 174.

[0212] Figure 24 (c) is a diagram showing the situation where a second touch operation is performed while the first touch operation is being performed. For example... Figure 24 (c) As shown in T300, touch information of the second point is added to the touch information management table 172 (M320). On the other hand, as Figure 24 As shown in D300 of (c), window information 174 stores "external window". In this case, the first touch operation and the second touch operation are processed as touch operations on the external window.

[0213] Figure 24 (d) is a diagram showing the case where, during a touch operation at the first point, the touched position moves to the area displaying internal content (the internal window). For example... Figure 24 As shown in T300 of (d), the touch information (M330) of the first point in the touch information management table 172 is updated, which stores the coordinates of the touch position after the movement and the action (“move”).

[0214] Figure 25 (a) is a diagram illustrating the case where, during a second-point touch operation, the touched position moves to an area displaying external content (the external window). For example... Figure 25 As shown in T300 of (a), the touch information (M340) of the second point in the touch information management table 172 is updated, which stores the coordinates of the touch position after the movement and the action (“move”).

[0215] in addition, Figure 24 (d) Touch operations based on touch information and Figure 25 In (a), all touch operations based on touch information are treated as touch operations on external windows.

[0216] Figure 25(b) is a diagram showing the state after all touch operations have ended. The touch information of the first point (M350) and the touch information of the second point (M352) are cleared, becoming... Figure 24 (a) The same state. At this point, if a new touch operation is performed, such as... Figure 25 As shown in (c), the touch information of the first point (M360) and the window where the first point was touched are stored in the touch information management table 172. Figure 25 (c) In the example, it is an "inner window".

[0217] Furthermore, when window information 174 is an "internal window," the internal window engine unit 106 processes touch operations based on touch information stored in the touch information management table 172. On the other hand, when window information 174 is an "external window," the internal window engine unit 106 notifies the browser control unit 100 of the touch information stored in the touch information management table 172. Since the touch information is notified from the browser control unit 110 to the external window engine unit 108 via the display control unit 104, the external window engine unit 108 processes touch operations based on the notified touch information.

[0218] In addition, the internal window engine unit 106 can also be set to drag and drop if the touch operation ends across windows after the touch operation has started, so that the information selected when the touch operation started can be transferred to other windows.

[0219] Thus, when a multi-touch operation is performed, the image forming apparatus of this embodiment can process a series of touch operations input from the start of the touch to the end of all touch operations as operations within the window corresponding to the touch position of the first point. Therefore, even when the touch position crosses the window via, for example, a swipe operation or an outward swipe operation, the image forming apparatus of this embodiment can also process the operation as an operation within the window corresponding to the position of the start touch operation.

[0220] [4. Fourth Implementation Method]

[0221] Next, the fourth embodiment will be described. The fourth embodiment is an implementation that manages multi-touch operations using a different method than the management of multi-touch operations in the third embodiment. This embodiment is a modification of the first embodiment. Figure 2 Replace with Figure 26 The first embodiment Figure 10 Replace with Figure 27 The figure is shown. Furthermore, the same reference numerals are used to label the same functional parts and processes, and descriptions are omitted.

[0222] In this embodiment, when a touch operation continues across a window, the touch operation performed before crossing the window is designated as the end of the touch, and the touch operation performed after crossing the window is designated as the start of the touch in the touched window. That is, in this embodiment, touches in each window are directly managed as the processing of each window.

[0223] [4.1 Functional Composition]

[0224] Reference Figure 26 This embodiment explains the functional configuration of the image forming apparatus 14. The image forming apparatus 14 and... Figure 2 Compared to the image forming apparatus 10 shown, the storage unit 160 also stores an internal window touch information management table 176 and an external window touch information management table 178. The information stored in the internal window touch information management table 176 and the external window touch information management table 178 is the same as that in the touch information management table 172 of the third embodiment.

[0225] [4.2 Processing Flow]

[0226] Reference Figure 27 The processing performed by the internal window engine unit 106 in this embodiment will be described. First, when the internal window engine unit 106 is notified of a touch event, it determines whether to update the touch information (step S400). The processing in step S400 is related to... Figure 23 The process in step S304 is the same.

[0227] If the touch information is not updated, the inner window engine unit 106 determines whether the touched location is inside the transparent area (step S400; no → step S402). If the touched location is not inside the transparent area, the inner window engine unit 106 adds touch information for the inner window (step S402; no → step S404). For example, the inner window engine unit 106 performs a process similar to... Figure 23 The same process is applied to steps S306 to S310, and the touch ID, coordinates, and action are stored in the touch information stored in the touch information management table 176 of the internal window where the touch presence or absence is "no". On the other hand, when the touched position is a transparent area, the internal window engine unit 106 adds touch information for the external window (step S402; yes → step S406). For example, the internal window engine unit 106 uses the same process as in step S404 to store the touch ID, coordinates, and action in the touch information of the touch information management table 178 of the external window where the touch presence or absence is "no".

[0228] On the other hand, the internal window engine unit 106 performs touch information update processing when updating touch information (step S400; Yes → step S408). The touch information update processing will be described next.

[0229] Next, the internal window engine unit 106 determines whether the touch information of the external window has been updated (step S410). For example, if touch information has been added to or updated in the touch information management table 178 for the external window, the internal window engine unit 106 determines that the touch information of the external window has been updated. If the touch information of the external window has been updated, the internal window engine unit 106 notifies the browser control unit 110 of the operation (touch event) based on the touch information stored in the touch information management table 178 for the external window as a touch event for the external window (step S410; Yes → step S412). At this time, the internal window engine unit 106 notifies the browser control unit 110 after subtracting the value corresponding to the height of the system area from the information relative to the Y coordinate. On the other hand, if the touch information of the external window has not been updated, the internal window engine unit 106 omits the processing in step S412 (step S410; No).

[0230] Furthermore, when touch information for an internal window exists, the internal window engine unit 106 processes touch operations based on that touch information as touch operations on the internal window (step S414; Yes → step S144). For example, the internal window engine unit 106 processes touch operations based on touch information stored in the internal window touch information management table 176 where the touch presence / absence is "Yes" as touch operations on the internal window. Conversely, when there is no touch information for an internal window (when the internal window touch information management table 176 does not store touch information where the touch presence / absence is "Yes"), the internal window engine unit 106 omits the processing in step S144 (step S414; No).

[0231] Next, refer to Figure 28 The process for handling touch information updates is explained below. First, the internal window engine unit 106 determines the touch information that is to be updated from the touch information stored in the internal window touch information management table 176 or the external window touch information management table 178 (step S450). Next, the internal window engine unit 106 determines whether the coordinates of the determined touch information before the update are coordinates within the visible area (step S452).

[0232] If the coordinates before the update are not within the transparent area, the internal window engine unit 106 determines whether the updated coordinates are within the transparent area (step S452; no → step S454). If the updated coordinates are not within the transparent area, the internal window engine unit 106 updates the touch information determined in step S450 based on the touch event notified in step S140 (step S454; no → step S456). In this case, the touch position does not change from outside the transparent area before and after the touch information update, therefore the touch information of the internal window is updated.

[0233] On the other hand, if the internal window engine unit 106 determines in step S454 that the updated coordinates are coordinates within the area, it clears the touch information (touch information of the internal window) determined in step S450 (step S454; yes → step S458). Furthermore, the internal window engine unit 106, through... Figure 27 The same process is applied to step S406 to add touch information for the external window (step S460). Thus, when the touch position of a touch operation on an area outside the visible area moves to the visible area, the internal window engine unit 106 can set the touch operation on the visible area as an operation on the external window.

[0234] Furthermore, if the inner window engine unit 106 determines in step S452 that the coordinates before the update are coordinates within the transparent area, it determines whether the updated coordinates are coordinates within the transparent area (step S452; yes → step S462). If the updated coordinates are coordinates within the transparent area, the inner window engine unit 106 updates the touch information determined in step S450 based on the touch event notified in step S140 (step S462; yes → step S464). In this case, the touch position does not change from within the transparent area before and after the touch information update, therefore the touch information of the outer window is updated.

[0235] On the other hand, if the internal window engine unit 106 determines in step S462 that the updated coordinates are not coordinates within the transparent area, it clears the touch information (touch information of the external window) determined in step S450 (step S462; no → step S466). Further, the internal window engine unit 106, through... Figure 27 The same process is applied to step S406 to add touch information for the inner window (step S468). Thus, when the touch position of a touch operation on the through area moves to an area outside the through area, the inner window engine unit 106 can treat the touch operation on the area outside the through area as an operation on the inner window.

[0236] [4.3 Action Examples]

[0237] Reference Figure 29 and Figure 30 Here is an example of the action in this implementation. Figure 29 as well as Figure 30 This diagram represents a display screen W400 containing an area E400 displaying an inner window, an area E402 displaying an outer window (the area through which the inner window is visible), a display T400 displaying content stored in the touch information management table 176, and a display T402 displaying content stored in the touch information management table 178. Furthermore, T400 and T402, from left to right, represent the touch number, touch presence / absence, X coordinate, Y coordinate, and action. Additionally, the numbers included in the display screen W400 correspond to the touch numbers of the touch information stored in the touch information management table of the corresponding area.

[0238] Figure 29 (a) is a diagram showing the case where no touch operation is performed. In the case where no touch operation is performed, the touch information stored in the inner window touch information management table 176 and the outer window touch information management table 178 is cleared.

[0239] Figure 29 (b) is a diagram showing the case where the first point of touch operation is performed on the external window. Figure 29 (b) As shown in T402, in the touch information management table 178 of the external window, the touch information of the first point is added as the touch information of touch number 1 (M410).

[0240] Figure 29 (c) is a diagram showing the situation where a second touch operation is performed on the inner window while the first touch operation is being performed. Figure 29 (c) As shown in T400, in the internal window touch information management table 176, the touch information of the second point is added as the touch information with touch number 1 (M420).

[0241] Figure 30 (a) is a diagram showing the case where the touch position of the touch operation managed by touch information with touch number 1 in the touch information management table 178 of the outer window is moved to the inner window (drag). When crossing windows from the outer window to the inner window, the touch operation of the outer window ends, the corresponding touch information is cleared from the touch information management table 178 of the outer window (M432) and appended to the touch information management table 176 of the inner window (M430). Furthermore, in Figure 30In (a), the touch information with touch number 2 is cleared from the touch information management table 176 in the inner window. Therefore, the touch information of the touch operation that moves the touch position to the inner window is managed as the second touch information of the inner window. As a result, the touch operation as the second point is processed after it begins in the inner window.

[0242] Figure 30 (b) is a diagram showing the end of the touch operation corresponding to touch information with touch number 2 in the internal window touch information management table 176. In this case, the corresponding touch information is cleared from the internal window touch information management table 176 (M440).

[0243] Figure 30 (c) is a diagram showing the case where the touch position of a touch operation managed as touch information with touch number 1 in the internal window touch information management table 176 is moved to the external window (drag). In this case, the corresponding touch information is cleared (M450) in the internal window touch information management table 176 and appended to the external window touch information management table 178.

[0244] Furthermore, the internal window engine unit 106 processes touch operations based on touch information stored in the internal window touch information management table 176. Additionally, the internal window engine unit 106 notifies the browser control unit 110 of touch information stored in the external window touch information management table 178. Since the touch information is notified from the browser control unit 110 to the external window engine unit 108 via the display control unit 104, the external window engine unit 108 processes touch operations based on the notified touch information.

[0245] Thus, when a touch operation is performed across a window, the image forming apparatus of this embodiment can process the touch operation as an operation within the window where the touch location is located.

[0246] [5. Variations]

[0247] This invention is not limited to the embodiments described above, and various modifications can be made. That is, embodiments obtained by appropriately changing and combining technical means without departing from the scope of this invention are also included within the technical scope of this invention. For example, the above embodiments can be extended to display two or more windows, and the security layer can be finely controlled for each window. In this case, the number of windows can be set to 3, and the native GUI can be displayed in the third window.

[0248] Furthermore, the above embodiments are described separately for ease of explanation, and can of course be implemented in combination within the scope of technical feasibility. For example, the second and third embodiments can also be combined. In this case, the image forming apparatus can display a native GUI and can appropriately handle multi-touch operations.

[0249] Furthermore, the programs operating in each device of this embodiment are programs that control the CPU, etc. (programs that enable the computer to perform its functions), in order to achieve the functions of the above-described embodiments. Moreover, the information processed by these devices is temporarily stored in a temporary storage device (e.g., RAM) during processing, and subsequently stored in various storage devices such as ROM (Read Only Memory), HDD, etc., and read, modified, and written by the CPU as needed.

[0250] Here, the storage medium for storing the program can be any of the following: semiconductor media (e.g., ROM, non-volatile memory cards, etc.), optical recording media / magneto-optical recording media (e.g., DVD (Digital Versatile Disc), MO (Magneto Optical Disc), MD (Mini Disc), CD (Compact Disc), BD (Blu-ray Disc), etc.), magnetic recording media (e.g., magnetic tape, floppy disk, etc.). Furthermore, by executing the downloaded program, not only can the functions of the above-described embodiments be achieved, but the functions of the present invention can also be realized through joint processing by an operating system or other applications based on the instructions of the program.

[0251] Furthermore, when circulating in the market, the program can be stored on a portable recording medium for distribution, or transmitted to a service timer via a network such as the Internet. In this case, the storage device of the server computer is naturally also included in this invention.

[0252] Explanation of reference numerals in the attached figures

[0253] 10, 12, 14 Image forming apparatus

[0254] 100 Control Department

[0255] 102 Image Processing Department

[0256] 104 Display Control Unit

[0257] 106 Internal Window Engine Department

[0258] 108 External Window Engine Department

[0259] 110 Browser Control Department

[0260] 112 HTTP Server Department

[0261] 120 Image Input Section

[0262] 130 Image forming unit

[0263] 132 Paper Supply Department

[0264] 134 Printing Department

[0265] 140 Display Section

[0266] 150 Operations Department

[0267] 160 Storage Department

[0268] 162 Operating System

[0269] 164 Web Browser Applications

[0270] 166 Browser Controller Application

[0271] 168 Content Data Storage Area

[0272] 170 Screen Setting Information Storage Area

[0273] 172 Touch Information Management Table

[0274] 174 Window Information

[0275] 176 Internal window touch information management table

[0276] 178 External Window Touch Information Management Table

[0277] 190 Ministry of Communications

[0278] 195 Power Supply Department

Claims

1. A display device, characterized in that, It has a display unit and a control unit. The control unit includes: An internal window engine unit controls a first display screen, the first display screen containing internal content displayed on the display unit; An external window engine unit controls a second display screen located behind the first display screen and including external content that overlaps with the first display screen and is displayed on the display unit; and The browser control unit communicates information between the internal window engine unit and the external window engine unit. The external window engine unit processes operations on the transparent area set in the first display screen as operations on the second display screen, and based on the operations on the second display screen, notifies the internal window engine unit of the input object's startup request via the browser control unit. The internal window engine unit processes operations on areas outside the transparent area as operations on the first display screen, and based on the input object's activation request, displays an input object on the first display screen for inputting external content. The browser control unit then notifies the external window engine unit of the response based on the operation on the input object. The external window engine unit reflects the result response in the external content.

2. The display device as claimed in claim 1, characterized in that, The display device operates in an environment where the display of internal and external content on a single screen is restricted due to safety constraints. The result response includes information indicating the selected button and information about the entered string.

3. The display device as claimed in claim 1, characterized in that, The input object is a soft keyboard. When the external window engine detects an operation on the string input field contained in the external content displayed on the second display screen, it notifies the internal window engine via the browser control unit of a soft keyboard activation request. Based on the launch request of the soft keyboard, the internal window engine displays the soft keyboard on the first display screen, and notifies the external window engine of the result response containing the string entered into the soft keyboard via the browser control unit. The external window engine will reflect the string contained in the result response in the string input field.

4. The display device as claimed in any one of claims 1 to 3, characterized in that, If the control unit performs other touch operations after starting a touch operation on the transparent area and before the touch operation ends, it processes the other touch operations and touch operations input in conjunction with those other touch operations as operations on the second display screen.

5. The display device as claimed in any one of claims 1 to 3, characterized in that, When the control unit moves a touch position from an area outside the transmission area to an area within the transmission area, it processes the touch operation on the transmission area as an operation on the second display screen.

6. A method for controlling a display device, executed using an internal window engine unit, an external window engine unit, and a browser control unit. The internal window engine controls the first display screen containing the internal content. The external window engine controls a second display screen, which is located behind the first display screen and includes external content that overlaps with the first display screen. The browser control unit communicates information between the internal window engine unit and the external window engine unit. The control method is characterized by including: The external window engine unit will process the operation of the transparent area set in the first display screen as a step for processing the operation of the second display screen; The internal window engine unit will process operations on areas outside the transparent area as operations on the first display screen; The external window engine unit notifies the internal window engine unit of the input object's startup request via the browser control unit based on the operation of the second display screen; the internal window engine unit displays an input object for inputting external content on the first display screen based on the input object's startup request. The step of the internal window engine unit responding to the result of the operation on the input object and notifying the external window engine unit via the browser control unit; as well as The step of the external window engine reflecting the result response to the external content.

Images