Information processing system, control method of information processing system, and program

Abstract

To provide an information processing system, a control method of the information processing system, and a program capable of quickly confirming input information when the input information is input to an image.SOLUTION: A cloud browser system 1000 includes an image formation device 10 and an image generation system 100. The image formation device 10 includes: input means (operation unit 312) capable of inputting input information to an image; device side generation means (browser control unit 360) which generates a composite image formed by combining an image of a prior stage before input of the input information and the input information having been input to the image; and control means (U control unit 352) which performs control to allow display means to display the composite image. The image generation system 100 generates image data of a reflection image in which the image is reflected to a state where the input information is input. A controller unit 300 implements control so as to replace the composite image with the reflection image after the composite image is displayed.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program.

Background Art

[0002] Conventionally, an image processing apparatus communicably connected to an external device such as a server via a network has been known. Patent Document 1 discloses an image processing apparatus equipped with a web browser module. This web browser module reads and displays information on various websites (home pages) on the Internet. Then, a web browser screen is displayed on the display (LCD) of the image processing apparatus.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an image processing apparatus, for example, a web browser screen enabling keyword search can be displayed. In this case, a keyword to be searched can be input in the input field of the web browser screen. However, it takes, for example, about 5 to 10 seconds from when a keyword is input in the input field until the web browser screen with the input keyword is displayed. This is because the generation of the web browser screen with the input keyword is performed not by the image processing apparatus but by an external device such as a server. Therefore, for the user, it becomes difficult to confirm the keyword input in the input field or to determine whether the keyword is correct until the web browser screen with the input keyword is displayed.

[0005] The present invention has been made in view of the above problems. An object of the present invention is to provide an information processing system, a control method for the information processing system, and a program that can quickly confirm input information when the input information is input to an image.

Means for Solving the Problems

[0006] In order to achieve the above object, an information processing system of the present invention is an information processing system including an information processing device and a server communicably connected to the information processing device, wherein the information processing device includes: display means for displaying image data from the server as an image; input means capable of inputting input information for the image displayed on the display means; device-side generation means for generating a composite image by combining the image in the state before the input information is input and the input information input by the input means for the image; device-side transmission means for transmitting the input information to the server; and control means for controlling to display the composite image on the display means. The server includes server-side reception means for receiving the input information from the device-side transmission means, server-side generation means for generating image data of a reflected image in which the image is reflected in a state where the input information is input based on the input information received by the server-side reception means, and server-side transmission means for transmitting the image data of the reflected image to the information processing device. The composite image is an image imitating the reflected image, and the control means performs control to replace the composite image with the reflected image after the composite image is displayed on the display means.

Effects of the Invention

[0007] According to the present invention, when input information is input to an image, the input information can be quickly confirmed.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 14C

Figure 15

Figure 16

Figure 17

Embodiments for Carrying Out the Invention

[0009] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration capable of exhibiting the same function. Also, an arbitrary component may be added. Further, any combination of two or more configurations (features) among the embodiments can also be made.

[0010] <First Embodiment> Hereinafter, a first embodiment will be described with reference to FIGS. 1 to 12. FIG. 1 is a block diagram showing the overall configuration of a cloud browser system according to the first embodiment. The cloud browser system 1000 shown in FIG. 1 is an information processing system capable of processing various types of information. The cloud browser system 1000 includes a plurality of image forming apparatuses (information processing apparatuses) 10 as information processing apparatuses and an image generation system 100, and these are connected to each other so as to be communicable. This cloud browser system 1000 is communicably connected to an external site 105 and an authentication site 110 via the Internet 111. Each image forming apparatus 10 is connected to the Internet 111 via a proxy server 104.

[0011] In this embodiment, the cloud browser system 1000 has three image forming apparatuses 10, but the number of arranged image forming apparatuses 10 is not limited to this, and for example, it may be one, two, or four or more. Hereinafter, each image forming apparatus 10 will be referred to as "image forming apparatus 101", "image forming apparatus 102", and "image forming apparatus 103", and among these image forming apparatuses 10, the image forming apparatus 101 will be representative. The cloud browser system 1000 has one image generation system 100, but the number of arranged image generation systems 100 is not limited to this, and for example, it may be two or more. Also, it is preferable that the number of arranged image generation systems 100 is less than the number of arranged image forming apparatuses 10. In the cloud browser system 1000, the proxy server 104 may be omitted. For example, assume that a user using the image forming apparatus 101 is browsing the screen displayed on the image forming apparatus 101. In this case, the user operates the image forming apparatus 101 to input a browsing request for the external site 105 to be browsed. Thereby, the image forming apparatus 101 transmits the address (URL) of the external site 105 to the image generation system 100 via the proxy server 104. The external site 105 is an http server that returns an http response to an http request, but its configuration is not particularly limited.

[0012] The image generation system 100 is a server having a gateway 106, a virtual machine 107, and a storage 109. On the virtual machine 107, a browser engine, which is a software module, operates. The browser engine receives the URL of the external site 105 sent from the image forming apparatus 101 via the gateway 106. Then, the browser engine accesses the external site 105 corresponding to this URL via the gateway 106 and receives web content such as HTML from the external site 105. Thereafter, a software module for performing rendering generates a rendering result of this web content. The rendering result is sent to the image forming apparatus 101 via the gateway 106. The image forming apparatus 101 receives the rendering result sent from the image generation system 100 and displays the rendering result. Thereby, the user can check the content of the external site 105. Here, "rendering" generally means displaying a browser screen (an image of the website supplied from the external site 105) based on web content. In the present embodiment, it refers to converting web content such as HTML received from the external site 105 into image data of a predetermined screen size.

[0013] The storage 109 stores user data such as cookies obtained by a browser engine operating on the virtual machine 107 interpreting Web content received from an external site 105. The cloud browser system 1000 is configured to be communicable with an authentication site 110. The authentication site 110 can identify a user who is using the image forming apparatus 10 (image forming apparatuses 101 to 103). The image generation system 100 obtains authentication of a user based on user information transmitted from the image forming apparatus 10. Then, the image generation system 100 manages the user data stored in the storage 109 for each authenticated user. In the present embodiment, the image generation system 100 has one virtual machine 107, but the number of virtual machines 107 arranged is not limited to this, and may be, for example, plural. In this case, it is preferable to arrange a load balancer between the gateway 106 and each virtual machine 107. Further, it is preferable to select one virtual machine 107 from among the plurality of virtual machines 107 according to the load situation. The storage 109 is preferably used together with the plurality of virtual machines 107.

[0014] FIG. 2A is a block diagram showing an example of the hardware configuration of a virtual machine. As shown in FIG. 2A, the virtual machine 107 has, as hardware, a CPU 201, an internal storage 202, a RAM 203, an interface 204, and a communication interface 205, which are connected via a bus 206. The CPU 201 executes various processes using computer programs and data stored in the internal storage 202. Thereby, the CPU 201 can control the operation of the entire virtual machine 107. The internal storage 202 stores setting data of the virtual machine 107, computer programs and data related to the startup of the virtual machine 107, computer programs and data related to the basic operation of the virtual machine 107, and the like. The RAM 203 has an area for storing computer programs and data loaded from the internal storage 202, data received via the communication interface 205, and the like. The RAM 203 also has a work area used when the CPU 201 executes various processes. In this way, the RAM 203 can appropriately provide various areas (storage areas). The interface 204 is an interface including a display unit for displaying the processing result by the CPU 201 as an image, characters, or the like, and an operation unit that the user operates to perform various operation inputs. The display unit includes, for example, a liquid crystal screen or a touch panel screen. The operation unit includes, for example, a keyboard, a mouse, a touch panel screen, and the like. The communication interface 205 is an interface for performing data communication with an external device. Specifically, the communication interface 205 can receive image data and the like from the image forming apparatus 10 (network I / F 310) (server-side reception step), and can transmit image data and the like to the image forming apparatus 10 (server-side transmission step). Thus, the communication interface 205 in the present embodiment has a function as a reception means (server-side reception means) for receiving image data and the like, and a function as a transmission means (server-side transmission means) for transmitting image data and the like. Note that in the virtual machine 107, the part functioning as the reception means and the part functioning as the transmission means may be provided separately.

[0015] Note that the hardware configuration of the virtual machine 107 is not limited to the configuration shown in FIG. 2A. For example, a memory device or the like may be further connected to the bus 206. The memory device is not particularly limited, and examples thereof include a hard disk drive, a USB memory, a magnetic card, an optical card, an IC card, a memory card, a flexible disk (FD), and a drive device for an optical disk such as a Compact Disk (CD). Further, the virtual machine 107 can be configured by virtualization technology, and various resources constituting the computer system can be organized into logical units independently of the physical configuration.

[0016] FIG. 2B is a block diagram showing an example of the software configuration of the virtual machine. As shown in FIG. 2B, the virtual machine 107 includes a browser processing unit 250 and an NW (Net Work) control unit 254 as software. The browser processing unit 250 includes an overall control unit 251, a browser engine 252, and a drawing control unit 253. The browser processing unit 250 is a module that is responsible for all processes related to the browser executed on the image generation system 100 side, which is the cloud. The overall control unit 251 is a module that controls the entire browser processing unit 250 operating on the virtual machine 107. When the user inputs a browsing request for an external site 105 that he / she wants to browse by operating the image forming apparatus 101, the URL of the external site 105 is transmitted from the image forming apparatus 10 to the image generation system 100 via the proxy server 104.

[0017] The overall control unit 251 collectively receives various requests transmitted from these image forming apparatuses 10. Further, the overall control unit 251 appropriately distributes the requests received from the image forming apparatuses 10 to the browser engine 252 and the drawing control unit 253. The browser engine 252 accesses the external site 105 via the NW control unit 254 based on the URL transmitted from the overall control unit 251. Thereby, the web content of the external site 105 can be acquired. Note that the browser engine 252 incorporates the function of a client of the HyperText Transfer Protocol (http) required to acquire the web content. Then, the browser engine 252 can appropriately store information such as cookies in the storage 109 by analyzing the web content, or refer to information such as cookies previously stored in the storage 109. Also, when the analysis of the web content is completed, the browser engine 252 requests the drawing control unit 253 to perform rendering processing. The drawing control unit 253 renders the web content based on the rendering request from the browser engine 252. Also, the drawing control unit 253 transmits the rendering result to the image forming apparatus 10 that is the request source of the URL via the NW control unit 254. The NW control unit 254 is a module that relays between the browser processing unit 250 and external devices such as the external site 105 and the authentication site 110 by controlling the communication interface 205. Also, the NW control unit 254 has a protocol stack for http communication and the like required for accessing the external site 105.

[0018] As described above, in the cloud browser system 1000, a user can input a browsing request for an external site 105 that he / she wants to operate and view using the image forming apparatus 101. As a result, the URL of the external site 105 is transmitted from the image forming apparatus 10 to the image generation system 100 via the proxy server 104. The overall control unit 251 receives the URL of the external site 105 and transfers it to the browser engine 252. As a result, the browser engine 252 receives this URL. Then, in order to acquire the web content indicated by the URL, the browser engine 252 requests the NW control unit 254 to access the external site 105 indicated by the URL. The NW control unit 254 that has received the access request controls the communication interface 205 to access the external site 105 via the gateway 106 and the Internet 111. The NW control unit 254 acquires the web content from the external site 105 and transfers it to the browser engine 252.

[0019] Based on the web content transferred from the NW control unit 254, the browser engine 252 transmits and receives user data such as cookies to and from the storage 109 as necessary, and analyzes the web content. In addition, the browser engine 252 requests the rendering control unit 253 to render the web content according to the analysis of the web content. The rendering control unit 253 that receives this request executes the rendering of the web content. When the rendering is completed, the rendering control unit 253 notifies the overall control unit 251 of the completion. The overall control unit 251 notifies the image forming apparatus 10 that is the source of the URL request of the completion of the rendering via the NW control unit 254. The image forming apparatus 10 that has received the completion of the rendering requests the image generation system 100 to acquire the rendering result. This request is transmitted to the overall control unit 251 via the NW control unit 254 and transferred to the rendering control unit 253 by the overall control unit 251. Then, the rendering control unit 253 that has received the request for acquiring the rendering result transmits the rendering result to the image forming apparatus 10 that is the source of the request via the NW control unit 254. As a result, the image forming apparatus 10 can display the rendering result. Then, the user can view the content of the desired external site 105 on the image forming apparatus 10.

[0020] Figure 3A is a block diagram showing an example of the hardware configuration of the image forming apparatus. As shown in Figure 3A, the image forming apparatus 10 includes, as hardware, a controller unit 300 (control means), an operation unit 312, a scanner 370, and a printer 395. The controller unit 300 includes a CPU 301, a RAM 302, a ROM 303, a storage (storage means) 304, an image bus I / F 305, an operation unit I / F 306, a network I / F 310, a USB host I / F 313, and an RTC 315. These are connected to be communicable with each other via a system bus 307. In addition, the controller unit 300 includes a device I / F 320, a scanner image processing unit 380, and a printer image processing unit 390. These are connected to be communicable with each other via an image bus 308. The system bus 307 and the image bus 308 are connected via the image bus I / F 305.

[0021] The controller unit 300 controls the operations of the operation unit 312, the scanner 370, and the printer 395 (control step). For example, the controller unit 300 can perform control for realizing a print function (copy function) such as printing out the image data read by the scanner 370, which is an image input device, by the printer 395, which is an image output device. The CPU 301 starts up the OS (Operating System) 351 (see FIG. 3B) according to the boot program stored in the ROM 303. The CPU 301 executes the programs stored in the storage 304 on this OS. The RAM 302 is used as the working area of the CPU 301. The RAM 302 provides a working area and also provides an image memory area for temporarily storing image data. In the data storage area (storage means) 108 of the storage 304, for example, programs and image data are stored. This program is not particularly limited, and examples include control programs for causing a computer to execute each part and each means of the cloud browser system 1000 (control method of the information processing system). Note that this control program may be stored in the internal storage 202 of the image generation system 100, or may be distributed and stored in the storage 304 of the image forming apparatus 10 and the internal storage 202 of the image generation system 100. Also, the image data is not particularly limited, and examples include images from the external site 105 acquired via the image generation system 100 and the like.

[0022] In this embodiment, the operation unit 312 is composed of a touch panel having a touch function. As a result, the operation unit 312 can display, for example, image data from the image generation system 100 as an image (display step). Note that the touch panel may be a pressure-sensitive touch panel or an electrostatic touch panel. Also, in the operation unit 312, when the user performs a touch operation on the touch panel, various types of information (input information) can be input to the image displayed on the touch panel (input step). As described above, the operation unit 312 of this embodiment has a function as a display means for displaying an image and a function as an input means for inputting various types of information. In the image forming apparatus 10, a part having a function as a display means and a part having a function as an input means may be provided separately. Also, in the operation unit 312, a moving operation for moving the image displayed on the touch panel can be performed by dragging (scrolling), that is, by sliding a fingertip on the touch panel. The operation unit I / F 306 is an interface responsible for connecting to the operation unit 312 and outputs image data to the operation unit 312. As a result, the operation unit 312 can display the image of the image data. Also, the operation unit I / F 306 transmits input information (operation information) input from the user via the operation unit 312 to the CPU 301.

[0023] The network I / F 310 is an interface for connecting the image forming apparatus 10 to a LAN. Thereby, the network I / F 310 can transmit, for example, input information or the like to the image generation system 100 (device-side transmission step). In addition, the network I / F 310 can also transmit the image data acquired by the scanner 370 to an external device. When the transmission of the image data is completed, the network I / F 310 can notify the image generation system 100 of the transmission completion information to that effect. Further, the network I / F 310 can also receive various information from the image generation system 100. Thus, the network I / F 310 in the present embodiment has a function as a receiving means (device-side receiving means) for receiving various information and a function as a transmitting means (device-side transmitting means) for transmitting various information. Note that in the image forming apparatus 10, the part functioning as the receiving means and the part functioning as the transmitting means may be provided separately. The USB host I / F 313 is an interface for communicating with the USB storage 314. Further, the USB host I / F 313 can transmit the data stored in the USB storage 314 to the CPU 301. Also, the USB host I / F 313 functions as an output unit for storing the data stored in the storage 304 in the USB storage 314. The USB storage 314 is an external storage device for storing data and is detachable from the USB host I / F 313. A plurality of USB storages 314 can be connected to the USB host I / F 313. The RTC 315 controls the current time. The time information controlled by this RTC 315 is used for recording the job input time and the like.

[0024] The image bus I / F 305 is a bus bridge that connects the system bus 307 and the image bus 308 for transferring image data at high speed and converts the data format. The image bus 308 is composed of a PCI bus or the like. A scanner 370 and a printer 395 are connected to the device I / F 320. The device I / F 320 performs synchronous / asynchronous conversion of image data. The scanner image processing unit 380 performs, for example, correction, processing, editing, etc. on the input image data. The printer image processing unit 390 performs, for example, correction according to the printer 395, resolution conversion, etc. on the print output image data. As described above, the image forming apparatus 10 is an apparatus having a print function and a scan function in this embodiment, but is not limited thereto, and may be an apparatus having at least one of a print function, a scan function, and a fax function.

[0025] FIG. 3B is a block diagram showing an example of the software configuration of the image forming apparatus. As shown in FIG. 3B, the image forming apparatus 10 includes, as software, a browser control unit 360, a UI control unit 352, a job execution control unit 353, a NW control unit 354, and a storage control unit 355. The browser control unit 360 includes an image data acquisition unit 361, a browser operation unit 362, a browser display unit 363, a command IF unit 364, and a proxy processing unit 365. Each unit surrounded by a solid line in FIG. 3B is a software module realized by the CPU 301 executing the main program loaded into the RAM 302. The main program is managed and controlled by the OS 351 for the execution of each module. The UI control unit 352 displays an image (screen) on the operation unit 312 and receives an operation from the user on the operation unit 312 via the operation unit I / F 306. Further, the UI control unit 352 can also send a notification to other modules, receive a drawing instruction from other modules, and control the screen update. The job execution control unit 353 is a module that receives a job execution instruction from the UI control unit 352 and controls job processing such as copy, scan, and print. In the present embodiment, when an operation for file selection is requested via the browser operation unit 362, an instruction to execute a scan job is sent from the browser operation unit 362 to the job execution control unit 353 via the UI control unit 352. The NW control unit 354 receives a communication request from other modules and controls communication with an external device by controlling the network I / F 310. Further, the NW control unit 354 can also receive a notification from the external device and notify the content of the notification to other modules. The storage control unit 355 records and manages the setting information and job information recorded in the storage 304. Each module located at the hierarchy of the OS 351 accesses the storage control unit 355 to refer to and set the setting values.

[0026] The browser control unit 360 is a sub-module included in the OS 351. When the browser operation unit 362 receives a notification of the user's operation from the UI control unit 352, it notifies the command IF unit 364 or the proxy processing unit 365 of the content of the user's operation. The proxy processing unit 365 receives the notification from the browser operation unit 362 and makes a request to the storage control unit 355 to obtain proxy setting information. Based on the proxy setting information acquired from the storage control unit 355, when the proxy setting is valid, the proxy processing unit 365 makes a communication request to the proxy server 104 via the NW control unit 354. Also, the proxy processing unit 365 receives a response to the communication request via the NW control unit 354. The proxy processing unit 365 notifies the browser display unit 363 or the command IF unit 364 of the result of processing the content of the response.

[0027] The command IF unit 364 receives notifications from the browser operation unit 362 and the proxy processing unit 365, and requests communication with the image generation system 100 via the NW control unit 354. The communication request at this time may include information notified from the browser operation unit 362 and the proxy processing unit 365. Further, this information includes user operation information such as text input, link button press and drag, and zoom. Also, the text input includes, for example, a URL. The link button press includes the press coordinates on the operation unit 312. The drag and zoom include a character string or the like associated with each operation. Also, the command IF unit 364 receives communication from the image generation system 100 via the NW control unit 354. The command IF unit 364 processes the communication content from the image generation system 100 and notifies the image data acquisition unit 361 or the browser display unit 363. The image data acquisition unit 361 receives a completion notification of the rendering result from the command IF unit 364. When the image data acquisition unit 361 receives the completion notification, it requests the image generation system 100 to acquire an image. The image acquisition request is transmitted to the drawing control unit 253 operating on the virtual machine 107 of the image generation system 100. Thereafter, the image is sent and the rendering result is transmitted to the image data acquisition unit 361. When the image data acquisition unit 361 receives the rendering result, it transfers the image, which is the rendering result, to the browser display unit 363. The browser display unit 363 receives the image from the image data acquisition unit 361 and instructs the UI control unit 352 to draw the image. Also, the browser display unit 363 receives notifications from the command IF unit 364 and the proxy processing unit 365, and instructs the UI control unit 352 to display a message corresponding to the notification.

[0028] FIG. 4 is a diagram showing an example of a home screen displayed on the operation unit of the image forming apparatus. The home screen 400 shown in FIG. 4 includes buttons 401, 402, 403, and 404. By operating, i.e., pressing, button 401, for example, it becomes possible to send an email to cloud A (not shown). By operating button 402, for example, it becomes possible to send an email to cloud B (not shown). By operating button 403, the browser control unit 360 is activated. By operating button 404, it becomes possible to perform a regular print on the printer 395. Thus, the home screen 400 includes four buttons for the user to instruct functions executable by the image forming apparatus 10. Note that the number of buttons included in the home screen 400 is not limited to four and can be changed as appropriate.

[0029] FIG. 5 is a diagram showing an example of a browser screen displayed on the operation unit of the image forming apparatus. The browser screen 500 shown in FIG. 5 is displayed when the browser control unit 360 is activated by operating button 403 on the home screen 400. The browser screen 500 includes a back button 501, a forward button 502, an address bar 503, a settings button 504, and a content area 505. Note that by operating the HOME key (not shown) installed on the operation unit 312, the activated browser control unit 360 can be forcibly terminated. The address bar 503 displays the URL "https: / / ***.***.***.*** / ". This URL is the address of the external site 105. In the content area 505, as a result of rendering the web content acquired from the external site 105, i.e., the screen of the website is displayed. Note that the content area 505 is configured such that the rendering result and the like are not displayed until the URL is input to the address bar 503 after the browser control unit 360 is activated. Also, on the browser screen 500, it may be possible to register the default URL with the items included in the settings button 504, and the result of rendering the web content of the URL may be displayed in the content area 505 simultaneously with the activation of the browser control unit 360.

[0030] FIG. 6A is a sequence diagram showing an example of processing executed among an image forming apparatus, an image generation system, and an external site. As shown in FIG. 6A, in step S601, a user who uses the image forming apparatus 10 operates (presses) a button 403 on a home screen 400 (see FIG. 4) displayed on an operation unit 312 of the image forming apparatus 10.

[0031] In step S602, the OS 351 of the image forming apparatus 10 requests the browser control unit 360 to start up. As a result, the browser control unit 360 starts up, and a browser screen 500 (see FIG. 5) is displayed on the operation unit 312 of the image forming apparatus 10.

[0032] In step S603a, the browser control unit 360 of the image forming apparatus 10 makes a rendering request for a web page to a browser processing unit 250 (virtual machine 107) of the image generation system 100. This rendering request includes, for example, URL information indicating the location of web content to be displayed on the browser screen 500, device information of the image forming apparatus 10, and the like. The "URL information" is information of a URL preset as a setting value of the button 403 or information of a URL input in the address bar 503 of the browser screen 500. The "device information" is, for example, screen size information of the operation unit 312 of the image forming apparatus 10, screen type information for specifying the operation unit 312, etc., but is not limited thereto.

[0033] In step S603b, the browser control unit 360 of the image forming apparatus 10 displays a loading screen on the operation unit 312. This loading screen is a screen for notifying the user that the processing is in progress until the browser screen 500 is displayed. Note that the notification screen is not limited to the loading screen and may be, for example, a popup or the like.

[0034] In step S604, when the browser processing unit 250 receives the rendering request in step S603a, it checks whether the user agent information stored in the storage 109 exists based on the device information in the rendering request.

[0035] In step S605, when there is user agent information in the storage 109, the browser processing unit 250 acquires the user agent information from the storage 109.

[0036] In step S606, based on the URL information acquired by the rendering request in step S603a and the user agent information acquired in step S605, the browser processing unit 250 makes a request to the external site 105 to acquire web content.

[0037] In step S607, the browser processing unit 250 receives web content from the external site 105 as a response to the web content acquisition request in step S606.

[0038] In step S608, the browser processing unit 250 renders the web content.

[0039] In step S609a, the browser processing unit 250 stores the result of the rendering performed in step S608 in the storage 109.

[0040] In step S609b, the browser processing unit 250 notifies the browser control unit 360 of the image forming apparatus 10 of the URL information indicating the location of the image data of the rendering result performed in step S608.

[0041] In step S610, the browser control unit 360 of the image forming apparatus 10 makes a request to the storage 109 of the image generation system 100 to acquire the image data that is the rendering result stored in step S609a. This request is made using the URL information notified in step S609b.

[0042] In step S611, the browser control unit 360 receives image data from the storage 109 as the acquisition request result of the image data in step S610.

[0043] In step S612, the browser control unit 360 gives an instruction to the OS 351 to display the image data received in step S611 in the content area 505 of the browser screen 500. As a result, a browser screen 500 including an image of the content area 505 suitable for the size of the operation unit 312, that is, the screen size of the touch panel, is displayed on the operation unit 312 of the image forming apparatus 10.

[0044] FIG. 6B is a sequence diagram showing an example of processing executed among the image forming apparatus, the image generation system, and the external site. As shown in FIG. 6B, in step S651, the user using the image forming apparatus 10 performs a tap operation on a URL (not shown) in the image displayed on the operation unit 312 of the image forming apparatus 10. The "tap operation (tap)" refers to an operation in which the user touches or presses an arbitrary location on the touch panel with a fingertip or the like. This image is an image in which the image data from the image generation system 100 is displayed as an image. Then, an electrical signal corresponding to the position of the fingertip (tap operation) on the touch panel is transmitted from the operation unit 312 to the UI control unit 352 (CPU 301). As a result, the UI control unit 352 can specify the coordinates of the position of the fingertip based on the electrical signal.

[0045] In step S652, the UI control unit 352 of the image forming apparatus 10 transmits operation information including information regarding the coordinates specified in step S651 to the browser control unit 360.

[0046] In step S653, the browser control unit 360 transmits the coordinate information (input coordinates) included in the operation information transmitted in step S652 to the browser processing unit 250 of the image generation system 100.

[0047] In step S654, the browser processing unit 250 analyzes whether the coordinates transmitted in step S653 indicate the position of the URL of the link destination, that is, whether there is a URL of the link destination at the coordinates. Here, as an example, it is assumed that as a result of analyzing the presence or absence of the URL of the link destination, there is a URL of the link destination.

[0048] In step S655, the browser processing unit 250 reflects the input by the tap operation in step S651, that is, it is determined that the user performed a tap operation on the URL in step S651.

[0049] In step S656, the browser processing unit 250 makes a request for the URL of the link destination, that is, a request for acquiring web content, to the external site 105 based on the analysis result in step S654.

[0050] In step S657, the browser processing unit 250 receives web content from the external site 105 as a response to the request for acquiring web content in step S656.

[0051] In step S658, the browser processing unit 250 performs rendering of the web content received in step S657.

[0052] In step S659a, the browser processing unit 250 stores the result of the rendering performed in step S658 in the storage 109.

[0053] In step S659b, the browser processing unit 250 notifies the browser control unit 360 of the image forming apparatus 10 of the URL information indicating the location of the image data of the result of the rendering stored in step S659a.

[0054] In step S660, the browser control unit 360 of the image forming apparatus 10 makes a request for acquiring image data to the storage 109 of the image generation system 100 based on the URL information notified in step S659b.

[0055] In step S661, the browser control unit 360 receives image data from the storage 109 as the acquisition request result of the image data in step S660.

[0056] In step S662, the browser control unit 360 gives an instruction to the UI control unit 352 to display the image data received in step S661. As a result, the image of the link destination analyzed in step S654 is displayed on the operation unit 312.

[0057] FIG. 7A is a sequence diagram showing an example of a conventional process executed among the image forming apparatus, the image generation system, and the external site. As shown in FIG. 7A, in step S701, the user performs a tap operation on an input field (see FIG. 11(a)) provided in the image of the website (hereinafter referred to as "website image") displayed on the operation unit 312 of the image forming apparatus 10. This website image is an image in which the image data from the image generation system 100 is displayed as an image. An example of the website image is the image shown in FIG. 11(a). This image will be described later. The "input field" is, for example, a field in which input information composed of characters (including numbers and symbols) can be input. The input information is not particularly limited, and examples thereof include keywords to be searched, etc. Then, the electrical signal in the input field is transmitted from the operation unit 312 to the UI control unit 352 (CPU 301). Thereby, the UI control unit 352 can specify the coordinates of the position of the fingertip based on the electrical signal.

[0058] In step S702, the UI control unit 352 of the image forming apparatus 10 transmits operation information including information regarding the coordinates specified in step S701 to the browser control unit 360.

[0059] In step S703, the browser control unit 360 transmits the coordinate information included in the operation information transmitted in step S702 to the browser processing unit 250 of the image generation system 100.

[0060] In step S704, the browser processing unit 250 analyzes whether the coordinates transmitted in step S703 indicate the position of an input field, that is, whether there is an input field at the coordinates. Here, as an example, it is assumed that as a result of analyzing the presence or absence of an input field, it is determined that there is an input field.

[0061] In step S705, the browser processing unit 250 requests the browser control unit 360 of the image forming apparatus 10 to input input information into the input field.

[0062] In step S706, the browser control unit 360 of the image forming apparatus 10 instructs the UI control unit 352 to display a software keyboard on the touch panel. As a result, the software keyboard is displayed and it becomes possible to input (receive) input information into the input field. As an example of the software keyboard, the image shown in FIG. 11(b) can be cited. This image will be described later. Note that the software keyboard and the input field may be displayed side by side on the touch panel, or the software keyboard may be displayed preferentially over the input field, that is, overlapping the input field.

[0063] In step S707, the user operates the software keyboard displayed in step S706 to input input information (characters) into the input field.

[0064] In step S708, the UI control unit 352 of the image forming apparatus 10 receives the input information input into the input field in step S707.

[0065] In step S709, the UI control unit 352 transmits the input information received in step S708 to the browser control unit 360. As a result, the browser control unit 360 can receive the input information.

[0066] In step S710, the browser control unit 360 transmits the input information received in step S709 to the browser processing unit 250 of the image generation system 100. As a result, the browser processing unit 250 can receive the input information.

[0067] In step S711, the browser processing unit 250 determines whether to reflect the input information received in step S710 in the input field of the website image.

[0068] In step S712, the browser processing unit 250 makes a request for the URL of the website image determined in step S711 to reflect the input information in the input field, that is, a request for acquiring web content, to the external site 105.

[0069] In step S713, the browser processing unit 250 receives web content from the external site 105 as a response to the web content acquisition request in step S712.

[0070] In step S714, the browser processing unit 250 performs rendering of the web content received in step S713. Specifically, the browser processing unit 250 (server-side generation means) generates image data of a reflected image in which the website image is reflected in a state where input information is input in the input field (server-side generation step). An example of the reflected image is the image shown in FIG. 12(b). This image will be described later.

[0071] In step S715a, the browser processing unit 250 stores the result of the rendering performed in step S714, that is, the image data of the reflected image, in the storage 109.

[0072] In step S715b, the browser processing unit 250 notifies the browser control unit 360 of the image forming apparatus 10 of the URL information indicating the location of the image data of the reflected image stored in step S715a.

[0073] In step S716, the browser control unit 360 of the image forming apparatus 10 makes a request to the storage 109 of the image generation system 100 to obtain the image data of the reflected image based on the URL information notified in step S715b.

[0074] In step S717, the browser control unit 360 receives the image data of the reflected image from the storage 109 as a result of the request for obtaining the image data in step S716.

[0075] In step S718, the browser control unit 360 gives an instruction to the UI control unit 352 to display the image data of the reflected image received in step S717. Thereby, the reflected image is displayed on the operation unit 312 instead of the composite image.

[0076] Conventionally, as described above, input information such as a keyword to be searched can be input in the input field of the website screen. However, after inputting the input information in the input field, it takes about 5 to 10 seconds or more depending on various conditions such as the communication speed and communication status until the reflected screen in the state where the input information is input is displayed. Therefore, for the user, it may be difficult to confirm the input information input in the input field until the reflected screen is displayed, or it may be difficult to determine whether the input information is correct.

[0077] Figure 7B is a flowchart showing the processing to be executed by the image generation system (browser processing unit). The flowchart shown in this Figure 7B is a flowchart showing the analysis processing in step S654 of the sequence diagram shown in Figure 6B and step S704 of the sequence diagram shown in Figure 7A. As shown in Figure 7B, in step S751, the browser processing unit 250 of the image generation system 100 performs analysis based on the coordinate information transmitted in step S653 or step S703. As a result of the analysis in step S751, if the coordinates are the coordinates indicating the position of the input field, the process proceeds to step S753. Also, as a result of the analysis in step S751, if the coordinates are the coordinates indicating the position of the URL of the link destination, the process proceeds to step S754. Also, as a result of the analysis in step S751, if the coordinates are neither the coordinates indicating the position of the input field nor the coordinates indicating the position of the URL of the link destination, the process ends.

[0078] In step S753, the browser processing unit 250 requests the browser control unit 360 of the image forming apparatus 10 to input input information to the input field. The processing in this step S753 is the same as the processing in step S705.

[0079] In step S754, the browser processing unit 250 determines that the input by the tap operation is to be reflected. The processing in this step S754 is the same as the processing in step S655.

[0080] Note that, as a result of the analysis in step S751, if the coordinates are neither the coordinates indicating the position of the input field nor the coordinates indicating the position of the URL of the link destination, that fact may be displayed on the operation unit 312.

[0081] As described above, conventionally, when input information such as a keyword to be searched is input into the input field of a website screen, it takes time until a reflection screen in the state where the input information is input is displayed. Therefore, for the user, there is a risk that it is difficult to confirm the input information input into the input field until the reflection screen is displayed. Thus, the cloud browser system 1000 is configured to be able to reduce such a phenomenon. Hereinafter, this configuration and operation will be described. FIG. 8 is a sequence diagram showing an example of processing executed among an image forming apparatus, an image generation system, and an external site. The sequence diagram shown in FIG. 8 is the same as the sequence diagram shown in FIG. 7A except mainly having steps S801 and S802. As shown in FIG. 8, after step S709 is executed, in step S801, the browser control unit 360 (apparatus-side generation means) of the image forming apparatus 10 generates a composite image (apparatus-side generation step). The "composite image" is an image obtained by combining a website image in a state before input information is input into the input field and the input information input into the input field of the website image. Also, the composite image is an image imitating a reflection image. An example of the composite image is the image shown in FIG. 12(a). This image will be described later.

[0082] In step S802, the browser control unit 360 gives an instruction to the UI control unit 352 to display the composite image generated in step S801. As a result, the composite image is displayed on the operation unit 312.

[0083] In step S718, the browser control unit 360 gives an instruction to the UI control unit 352 to display the image data of the reflected image received in step S717. As a result, on the operation unit 312, the composite image displayed in step S802 is replaced with the reflected image, that is, the display is changed from the composite image to the reflected image. Therefore, the user can view the composite image until the reflected image is displayed. In this way, the composite image functions as a connecting image until the reflected image is displayed. Thereby, the user can check the input information input to the input field and determine whether the input information is correct. As described above, in the cloud browser system 1000, when input information is input to the input field of the website image, the input information can be quickly checked. Note that in step S705 of the sequence diagram shown in FIG. 8, the browser processing unit 250 sends an input request for the input information to the input field to the browser control unit 360, and sends input-related information together with this request. The input-related information will be described later with reference to FIGS. 9 and 10. Also, in the sequence diagram shown in FIG. 8, the processing order is in the order of step S710 and step S801, but it is not limited to this. For example, the processing order may be in the order of step S801 and step S710.

[0084] FIG. 9 is a diagram showing an example of a website image displayed on the operation unit of the image forming apparatus. The website image 900 shown in FIG. 9 includes a name 901, an input field 902, and a button 903. The name 901 is the name of the website image 900. Input information can be input to the input field 902. Here, as an example, XY coordinates with the leftmost upper point (pixel) in the website image 900 as the origin O900 are assumed. The unit of size in the XY coordinates is, for example, a pixel or the like. The input field 902 forms a horizontally long rectangle, and the coordinates of the leftmost upper point are (X1, Y1), the coordinates of the rightmost upper point are (X2, Y1), the coordinates of the leftmost lower point are (X1, Y2), and the coordinates of the rightmost lower point are (X2, Y2). The button 903 is operated (pressed) in a state where input information is input to the input field 902. Thereby, a search result using the input information as a keyword is displayed.

[0085] FIG. 10 is a diagram for explaining input-related information regarding an input field of a website image. As shown in FIG. 10, the input-related information 1A includes input field information 1B and input content information 1C. Note that the input-related information 1A is not limited to the information shown in FIG. 10. Further, the input field information 1B includes, for example, position coordinates 1001 and size 1002. The position coordinates 1001 are the position coordinates of the input field 902, for example, (X1, Y1) described above. The size 1002 is the width and height (size) of the input field 902. The width of the input field 902 is obtained from (X2 - X1). The height of the input field 902 is obtained from (Y2 - Y1). Note that the size 1002 is not limited to being obtained from absolute coordinates, and may be obtained from relative coordinates corresponding to a percentage (%) of the entire website image 900, for example. The input content information 1C conforms to the information of HTML acquired from the external site 105. This. The input content information 1C includes, for example, input type specification 1003, character count limit 1004, character limit 1005, input mode 1006, and font 1007. The input type specification 1003 is a specification of what can be used for input to the input field 902. What can be used for input to the input field 902 is not particularly limited, and examples include a software keyboard and the like. The character count limit 1004 is the maximum number of characters that can be input within the input field 902. The character limit 1005 is the type of characters that can be used within the input field 902. The input mode 1006 is a mode that enables input to the input field 902. The font 1007 is the font of the characters that can be used within the input field 902. The input type specification 1003 to the input mode 1006 are usage conditions for the software keyboard.

[0086] FIG. 11 is a diagram showing an example of an image displayed on the operation unit of the image forming apparatus. FIG. 11(a) is a diagram showing an example of a website image. FIG. 11(b) is a diagram showing an example of a software keyboard. As shown in FIG. 11(a), on the operation unit 312, a website image 900, a first information image 1114, and a second information image 1115 are displayed. The website image 900 is an image including a name 901, an input field 902, and a button 903 as described above. In the state shown in FIG. 11(a), the input field 902 is blank and no input information has been entered yet. The first information image 1114 includes a forward button 1111 for performing an operation to advance the website image 900 to the next image and a back button 1112 for performing an operation to return the website image 900 to the original image. Also, the first information image 1114 includes an address bar 1113 indicating the URL of the link destination of the website image 900. A comment "Status Check" is displayed on the second information image 1115.

[0087] As shown in FIG. 11(b), on the operation unit 312, a software keyboard 1116, a first information image 1117, and a second information image 1115 are displayed. This software keyboard 1116 includes a key group 1118 and a display field 1119. The key group 1118 is composed of a plurality of keys such as alphabets and numbers, for example. In the display field 1119, the result input by operating any key of the key group 1118 is displayed. As an example of this result, "Word" is displayed in FIG. 11(b). A comment "INPUT" for prompting input by operating the software keyboard 1116 is displayed on the first information image 1117.

[0088] FIG. 12 is a diagram showing an example of an image displayed on the operation unit of the image forming apparatus. FIG. 12(a) is a diagram showing an example of a composite image. FIG. 12(b) is a diagram showing an example of a reflected image. As shown in FIG. 12(a), on the operation unit 312, a composite image 1200, a first information image 1214, and a second information image 1215 are displayed. In the composite image 1200, there are included a name 1201 imitating the name 901 of the website image 900, an input field 1202 imitating the input field 902 of the website image 900, and a button 1203 imitating the button 903 of the website image 900. In the input field 1202, input information (here, "Word") input with the software keyboard 1116 is displayed. Thus, the composite image 1200 is an image obtained by combining the website image 900 in the state before the input information is input into the input field 902 and the input information input to the input field 902. Note that the generation of the composite image 1200 is performed by the browser control unit 360 of the image forming apparatus 10. Also, the input information displayed in the input field 1202 may not match the input information input to the input field 902. This is because the input information displayed in the input field 1202 does not have to match the input information input to the input field 902 as long as the user can confirm the input content.

[0089] Also, when generating the composite image 1200, the browser control unit 360 preferably uses the same font as the font used in the image data of the website image 900 or a font similar to the font for the characters. For example, when it is impossible to use the same font as the font used in the image data of the website image 900, the browser control unit 360 preferably uses a font similar to the font used in the image data of the website image 900. More specifically, when it is impossible to use the same font as the font "Times New Roman" used in the image data of the website image 900, it is preferable to use the font "Times" similar to the font. Thereby, the composite image 1200 can be made as close as possible to the reflected image 900A described later, and the user can feel that they have smoothly moved, that is, transitioned, from the composite image 1200 to the composite image 1200. The first information image 1214 includes a button 1211 imitating the button 1111 of the first information image 1114, a button 1212 imitating the button 1112 of the first information image 1114, and an address bar 1213 imitating the address bar 1113 of the first information image 1114. The second information image 1215 is an image imitating the second information image 1115, and a comment "Status Check" is displayed in the same manner as the second information image 1115.

[0090] As shown in FIG. 12(b), the reflected image 900A, the first information image 1114, and the second information image 1115 are displayed. The reflected image 900A is an image obtained by reflecting the website image 900 in a state where input information (here, "Word") is input in the input field 902. Note that the generation of the reflected image 900A is performed by the browser processing unit 250 of the image generation system 100. Also, in the input field 902, information related to the leading input information "Word" is also displayed as a search candidate.

[0091] As described above, on the operation unit 312, the website image 900, the software keyboard 1116, the composite image 1200, and the reflected image 900A are displayed in this order. As a result, the image forming apparatus 10 can allow the user to visually recognize the composite image 1200 until the reflected image 900A is displayed, and thus can quickly confirm the input information input in the input field 902 of the website image 900.

[0092] <Second Embodiment> Hereinafter, with reference to FIGS. 13 to 17, the second embodiment will be described. The description will focus on the differences from the above-described embodiments, and the description of the same matters will be omitted. FIG. 13 is a sequence diagram showing an example of processing executed among the image forming apparatus, the image generation system, and the external site of the cloud browser system according to the second embodiment. In the sequence diagram shown in FIG. 13, the processing from step S710 to step S715b is the same as the processing from step S710 to step S715b in the sequence diagram shown in FIG. 8. Although not shown, in the sequence diagram shown in FIG. 13, the processing before step S710 is the same as the processing from step S701 to step S709 in the sequence diagram shown in FIG. 8. Also, the processing of step S801 and step S802 in the sequence diagram shown in FIG. 13 is the same as the processing of step S801 and step S802 in the sequence diagram shown in FIG. 8.

[0093] As shown in FIG. 13, in step S1301, the user who uses the image forming apparatus 10 performs a tap operation (pressing) on the input field in the composite image displayed on the operation unit 312 of the image forming apparatus 10. Note that this composite image is displayed on the touch panel (operation unit 312) in step S802. Then, due to the tap operation, the electrical signal in the input field is transmitted from the operation unit 312 to the UI control unit 352. As a result, the UI control unit 352 can specify the coordinates of the position of the fingertip based on the electrical signal.

[0094] In step S1302, the UI control unit 352 of the image forming apparatus 10 transmits operation information including information regarding the coordinates specified in step S1301 to the browser control unit 360.

[0095] In step S1303, the browser control unit 360 determines whether the tap operation in step S1301 is a tap operation on an input field. This determination is made based on the coordinates in the tap operation and the input field information 1B (see FIG. 10). For example, if the coordinates in the tap operation are located within the range of the input field 902 (see FIG. 9), it is determined that the tap operation in step S1301 is a tap operation on the input field. On the other hand, if the coordinates in the tap operation are located outside the range of the input field 902, it is determined that the tap operation in step S1301 is not a tap operation on the input field. Here, as an example, it is assumed that the tap operation in step S1301 is determined to be a tap operation on the input field.

[0096] In step S1304, the browser control unit 360 gives an instruction to the UI control unit 352 to display a software keyboard on the touch panel.

[0097] In step S1305, the user operates the software keyboard displayed in step S1304 to input the input information to the input field again. Note that the user has already input the input information once in step S707.

[0098] In step S1306, the UI control unit 352 of the image forming apparatus 10 accepts the input information input to the input field again in step S1305.

[0099] In step S1307, the UI control unit 352 transmits the re-input information accepted in step S1306 to the browser control unit 360. As a result, the browser control unit 360 can receive the re-input information.

[0100] In step S1308, the browser control unit 360 transmits the re - input information received in step S1307 to the browser processing unit 250 of the image generation system 100. As a result, the browser processing unit 250 can receive the re - input information.

[0101] In step S1309, the browser control unit 360 generates a composite image by combining the website image in the state before input information is input into the input field and the re - input information input to the input field of the website image in step S1305.

[0102] In step S1310, the browser control unit 360 gives an instruction to the UI control unit 352 to display the composite image generated in step S1309. As a result, the composite image generated in step S1309 is displayed on the operation unit 312.

[0103] In step S1311, the browser processing unit 250 determines whether to reflect the re - input information received in step S1308 in the state where the re - input information is input into the input field of the website image.

[0104] In step S1312, the browser processing unit 250 makes a request to the external site 105 for obtaining the URL (Web content) of the website image determined in step S1311 to reflect the re - input information in the input field.

[0105] In step S1313, the browser processing unit 250 receives Web content from the external site 105 as a response to the Web content acquisition request in step S1312.

[0106] In step S1314, the browser processing unit 250 performs rendering of the Web content received in step S1313. Specifically, the browser processing unit 250 generates image data of a reflected image in which the website image is reflected in the state where re - input information is input into the input field.

[0107] In step S1315a, the browser processing unit 250 stores the image data of the reflected image as the result of the rendering performed in step S1314 in the storage 109.

[0108] In step S1315b, the browser processing unit 250 notifies the browser control unit 360 of the image forming apparatus 10 of the URL information indicating the location of the image data of the reflected image stored in step S1315a.

[0109] In step S1316, based on the URL information notified in step S1315b, the browser control unit 360 of the image forming apparatus 10 makes a request to acquire the image data of the reflected image from the storage 109 of the image generation system 100.

[0110] In step S1317, the browser control unit 360 receives the image data of the reflected image from the storage 109 as the result of the acquisition request for the image data in step S1316.

[0111] In step S1318, the browser control unit 360 gives an instruction to the UI control unit 352 to display the image data of the reflected image received in step S1317. As a result, the reflected image is displayed on the operation unit 312 instead of the composite image. This reflected image is an image in a state where re-input information is input in the input field.

[0112] As described above, when a tap operation is performed on the input field in the composite image while the composite image is being displayed, the image forming apparatus 10 is configured to enable re-input of the input information to the input field. Thus, even if the input information is input incorrectly, the input information can be correctly re-entered. In step S715b, the URL information indicating the location of the image data of the reflected image is notified, but the subsequent request to acquire the image data of the reflected image based on the URL information is not made. Also, the reflected image may be displayed between step S802 and step S1301. In this case, if the timing of re-inputting the input information and the timing of displaying the reflected image are the same, it is preferable that the re-input of the input information is prioritized.

[0113] Next, a case where a tap operation is performed on a portion other than the input field in the composite image while the composite image is being displayed will be described. FIG. 14A is a sequence diagram showing an example of processing executed among the image forming apparatus, the image generation system, and the external site. In the sequence diagram shown in FIG. 14A, the processing from step S710 to step S718 is the same as the processing from step S710 to step S718 in the sequence diagram shown in FIG. 8. Although illustration is omitted, in the sequence diagram shown in FIG. 14A, the processing prior to step S710 is the same as the processing from step S701 to step S709 in the sequence diagram shown in FIG. 8. Also, the processing of step S801 and step S802 in the sequence diagram shown in FIG. 14A is the same as the processing of step S801 and step S802 in the sequence diagram shown in FIG. 8.

[0114] As shown in FIG. 14A, in step S1401, the user who uses the image forming apparatus 10 performs a tap operation on a portion other than the input field (for example, the background portion) in the composite image displayed on the operation unit 312 of the image forming apparatus 10. Then, an electrical signal in the portion other than the input field is transmitted from the operation unit 312 to the UI control unit 352. Thereby, the UI control unit 352 can specify the coordinates of the position of the fingertip based on the electrical signal.

[0115] In step S1402, the UI control unit 352 of the image forming apparatus 10 transmits operation information including information regarding the coordinates specified in step S1401 to the browser control unit 360.

[0116] In step S1403, the browser control unit 360 determines whether the tap operation in step S1401 is a tap operation on the input field. Here, it is assumed that the tap operation in step S1401 is determined to be a tap operation on the background portion.

[0117] In step S1404, the browser control unit 360 gives an instruction to the UI control unit 352 to display on the touch panel an image (hereinafter sometimes referred to as a "warning image") for warning that an image is being loaded. As a result, on the touch panel, the warning image is displayed instead of the composite image. An example of the warning image is the image shown in FIG. 15(a). This image will be described later. And in a state where the warning image is displayed, inputs other than the input of input information on the touch panel are restricted.

[0118] Also, in step S718, instead of the warning image, a reflected image is displayed.

[0119] As described above, when a tap is performed on a portion other than the input field in the composite image while the composite image is being displayed by the image forming apparatus 10, inputs other than the input of input information on the touch panel can be restricted until the composite image is replaced by the reflected image. This configuration is a preferable configuration when it is desired to restrict inputs other than the input of input information until the composite image is replaced by the reflected image.

[0120] FIG. 14B is a sequence diagram showing an example of processing executed among the image forming apparatus, the image generation system, and the external site. In the sequence diagram shown in FIG. 14B, the processing from step S710 to step S715b is the same as the processing from step S710 to step S715b in the sequence diagram shown in FIG. 8. Although illustration is omitted, in the sequence diagram shown in FIG. 14B, the processing before step S710 is the same as the processing from step S701 to step S709 in the sequence diagram shown in FIG. 8. Also, the processing of step S801 and step S802 in the sequence diagram shown in FIG. 14B is the same as the processing of step S801 and step S802 in the sequence diagram shown in FIG. 8.

[0121] As shown in FIG. 14B, in step S1451, the user who uses the image forming apparatus 10 performs a tap operation on a portion other than the input field (for example, button 1203) in the composite image displayed on the operation unit 312 of the image forming apparatus 10. Then, an electrical signal in the portion other than the input field is transmitted from the operation unit 312 to the UI control unit 352. As a result, the UI control unit 352 can specify the coordinates of the position of the fingertip based on the electrical signal.

[0122] In step S1452, the UI control unit 352 of the image forming apparatus 10 transmits operation information including information regarding the coordinates specified in step S1451 to the browser control unit 360.

[0123] In step S1453a, the browser control unit 360 determines whether the tap operation in step S1451 is a tap operation on the input field. Here, it is assumed that the tap operation in step S1451 is determined to be a tap operation on a portion other than the input field.

[0124] In step S1453b, the browser control unit 360 transmits the coordinate information (input coordinates) included in the operation information in step S1451 to the browser processing unit 250 of the image generation system 100.

[0125] In step S1454, the browser processing unit 250 analyzes whether the coordinates are the coordinates indicating the position of button 1203 based on the coordinate information transmitted in step S1453b. Here, as an example, it is assumed that the coordinates are the coordinates indicating the position of button 1203.

[0126] In step S1455, the browser processing unit 250 determines that the content of the input information is to be reflected.

[0127] In step S1456, the browser processing unit 250 makes a request for the URL of the link destination of the content of the input information determined to be reflected in step S1455, that is, a request for acquiring web content, to the external site 105.

[0128] In step S1457, the browser processing unit 250 receives web content from the external site 105 as a response to the acquisition request of the web content in step S1456.

[0129] In step S1458, the browser processing unit 250 performs rendering of the web content received in step S1457.

[0130] In step S1459a, the browser processing unit 250 stores the result of the rendering performed in step S1458 in the storage 109.

[0131] In step S1459b, the browser processing unit 250 notifies the browser control unit 360 of the image forming apparatus 10 of the URL information indicating the location of the image data of the rendering result stored in step S1459a.

[0132] In step S1460, the browser control unit 360 of the image forming apparatus 10 makes an acquisition request for the image data to the storage 109 of the image generation system 100 based on the URL information notified in step S1459b.

[0133] In step S1461, the browser control unit 360 receives the image data from the storage 109 as a result of the acquisition request of the image data in step S1460.

[0134] In step S1462, the browser control unit 360 gives an instruction to the UI control unit 352 to display the image data received in step S1461. As a result, an image (hereinafter sometimes referred to as an "input information image") indicating the content of the input information is displayed on the operation unit 312. An example of the input information image is the image shown in FIG. 15(b). This image will be described later.

[0135] As described above, when a tap operation is performed on a portion other than the input field in the composite image while the composite image is being displayed, the image forming apparatus 10 is configured such that the specific content of the input information can be displayed prior to the replacement of the composite image with the reflected image. This configuration is a preferable configuration when it is desired to display the specific content of the input information with higher priority than the reflected image.

[0136] FIG. 14C is a flowchart showing the processing to be executed by the image forming apparatus (browser control unit). The flowchart shown in FIG. 14C(a) is used for the determination process in step S1403 of the flowchart shown in FIG. 14A. The flowchart shown in FIG. 14C(b) is used for the determination process in step S1453a of the flowchart shown in FIG. 14B. As shown in FIG. 14C(a), in step S1481, the browser control unit 360 of the image forming apparatus 10 determines whether the tap operation is a tap operation on the input field. If, as a result of the determination in step S1481, it is determined that the tap operation is a tap operation on the input field, the process proceeds to step S1482. On the other hand, if, as a result of the determination in step S1481, it is determined that the tap operation is not a tap operation on the input field, the process proceeds to step S1483.

[0137] In step S1482, the browser control unit 360 gives an instruction to the UI control unit 352 to display (activate) the software keyboard. This step S1482 corresponds to step S706.

[0138] In step S1483, the browser control unit 360 gives an instruction to the UI control unit 352 to display a warning image on the touch panel. This step S1483 corresponds to step S1404.

[0139] The flowchart shown in FIG. 14C(b) is the same as the flowchart shown in FIG. 14C(a), except that the processing is different when it is determined in step S1481 that the tap operation is not on the input field. As shown in FIG. 14C(b), when it is determined in step S1481 that the tap operation is not on the input field, the process proceeds to step S1491. In step S1491, the browser control unit 360 transmits the coordinate information (input coordinates) obtained by the tap operation on the portion other than the input field in the composite image to the image generation system 100. This step S1491 corresponds to step S1453b.

[0140] FIG. 15 is a diagram showing an example of an image displayed on the operation unit of the image forming apparatus. FIG. 15(a) is a diagram showing an example of a warning image. FIG. 15(b) is a diagram showing an example of an input information image. As shown in FIG. 15(a), on the operation unit 312, a warning image 1500, the first information image 1214 and the second information image 1215 described above are displayed. The warning image 1500 is an image obtained, for example, by editing the composite image 1200 with the browser control unit 360. This warning image 1500 includes a pop-up 1501, an input field 1202, and a button 1203. In the pop-up 1501, a message indicating that it is being loaded (here, "Loading now. Please wait.") is displayed. Note that the display indicating that it is being loaded is not limited to the display using the pop-up 1501. As shown in FIG. 15(b), on the operation unit 312, an input information image 1511, the first information image 1114 and the second information image 1115 described above are displayed. The input information image 1511 includes input information content 1512, the name 901, the input field 902, and the button 903 described above. The input field 902 is in a state where input information (here, "Word") has been input. In the input information content 1512, a message indicating that it is a site explaining the input information "Word" is displayed. Note that in the input information content 1512, instead of this message, specific descriptions such as a text or a drawing explaining the input information "Word" may be displayed.

[0141] FIG. 16 is a diagram showing an example of an image displayed on the operation unit of the image forming apparatus. As shown in FIG. 16, a website image 1600, the aforementioned first information image 1114, and second information image 1115 are displayed on the operation unit 312. The website image 1600 includes an input field 1601 and a message 1602 prompting the input of a PIN code (input information) into the input field 1601. This website image 1600 has a change event preset. The "change event" means that after the input of the PIN code into the input field 1601 is completed, without performing a tap operation on, for example, a "Determine (GO)" button or the like, a transition is made to another website image different from the website image 1600. Here, the PIN code input into the input field 1601 is set to "987654".

[0142] FIG. 17 is a diagram showing an example of an image displayed on the operation unit of the image forming apparatus. FIG. 17(a) is a diagram showing an example of a composite image. FIG. 17(b) is a diagram showing an example of a warning image. Each image is an image displayed during the transition to another website image. As shown in FIG. 17(a), on the operation unit 312, a composite image 1700 imitating the website image 1600, the aforementioned first information image 1214, and the second information image 1215 are displayed. The composite image 1700 includes an input field 1701 imitating the input field 1601 and a pop-up 1702. In the input field 1701, the PIN code "987654" is entered. By checking the input field 1701, the user can grasp that the PIN code "987654" has been entered in the input field 1601. Also, in the pop-up 1702, a message indicating that the transition to another website image is in progress is displayed. By checking the pop-up 1702, the user can grasp that the image transition is in progress. As shown in FIG. 17(b), on the operation unit 312, a warning image 1711 that is not a composite image, the aforementioned first information image 1114, and the second information image 1115 are displayed. The warning image 1711 includes a pop-up 1712. In the pop-up 1712, a message indicating that the PIN code "987654" has been entered in the input field 1601 and that the transition to another website image is in progress is displayed. By checking the pop-up 1712, the user can grasp that the PIN code "987654" has been entered in the input field 1601 and that the image transition is in progress.

[0143] As described above, the preferred embodiments of the present invention have been explained. However, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist thereof. The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors of a computer of the system or apparatus to read and execute the program. Further, the present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions. Further, as the information processing apparatus, in each of the above-described embodiments, it is the image forming apparatus 10, but it is not limited thereto, and for example, a desktop type or notebook type personal computer, a smartphone, etc. may be used.

[0144] Further, in the cloud browser system 1000, for example, the image generation system 100 which is a server may exist outside Japan, and the image forming apparatus 10 which is a terminal device may exist within Japan. Even in this case, each file and data can be transmitted from the server to the terminal device, and the terminal device can receive each file and data. Thus, even when the server exists outside Japan, the transmission and reception (transmission and reception) of files and data in the cloud browser system 1000 can be performed integrally, that is, without being mediated by a separate operation by the user of the terminal device. In the cloud browser system 1000, for example, even when the server exists outside Japan and the terminal device exists within Japan, the terminal device can perform the main functions of the cloud browser system 1000. Further, the effects by the functions can be manifested within Japan. For example, even if the server exists outside Japan, if the terminal device constituting the cloud browser system 1000 exists within Japan, it is possible to use the system within the country using the terminal device.

[0145] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) An information processing system including an information processing apparatus and a server communicably connected to the information processing apparatus, The information processing device includes: a display unit that displays the image data from the server as an image; an input means capable of inputting input information for the image displayed on the display means; a device-side generating means for generating a composite image by combining the image in a state before the input information is input with the input information inputted by the input means; a device side transmitting means for transmitting the input information to the server; a control means for controlling the display means to display the composite image, The server, a server-side receiving means for receiving the input information from the device-side transmitting means; a server-side generating means for generating image data of a reflected image by reflecting the image in a state in which the input information was input based on the input information received by the server-side receiving means; a server-side transmitting means for transmitting image data of the reflected image to the information processing device; the composite image is an image that mimics the reflected image, The information processing system is characterized in that the control means performs control to replace the composite image with the reflection image after the composite image is displayed on the display means. (Configuration 2) The input information includes characters, The information processing system according to configuration 1, wherein the device-side generation means, when generating the composite image, uses a font for the characters that is the same as or similar to a font used in the image data. (Configuration 3) The information processing system described in Configuration 2, wherein the device-side generation means uses a font that is similar to the font used in the image data when it is impossible to use the same font as the font used in the image data. (Configuration 4) When the image displayed on the display means transitions to another image different from the said image upon completion of input of the input information by the input means, the display means displays a message indicating that the other image is in the process of transitioning. The information processing system according to any one of Configurations 1 to 3, characterized in that. (Configuration 5) The information processing system according to Configuration 4, characterized in that the display means displays a part of the composite image together with the message. (Configuration 6) The information processing system according to Configuration 4, characterized in that the display means displays the input information together with the message. (Configuration 7) An input field for inputting the input information is provided in the image, The device-side generation means generates, as the composite image, an image in a state where the input information is input in the input field, The information processing system according to any one of Configurations 1 to 6, characterized in that the server-side generation means generates, as the reflected image, an image in a state where the input information is input in the input field. (Configuration 8) The display means has a touch panel having a touch function, The control means, when a tap is performed on the touch panel with respect to the input field in the composite image in a state where the composite image is displayed on the display means, performs control to enable re-input of the input information by the input means. The information processing system according to Configuration 7, characterized in that. (Configuration 9) The display means has a touch panel having a touch function, The control means, when a tap is performed on the touch panel with respect to a part other than the input field in the composite image in a state where the composite image is displayed on the display means, performs control to display the specific content of the input information on the display means. The information processing system according to Configuration 7, characterized in that. (Configuration 10) The information processing system according to Configuration 9, characterized in that the control means performs control to display the specific content of the input information on the display means prior to control to replace the composite image with the reflected image. (Configuration 11) The display means has a touch panel with a touch function, The control means, in a state where the composite image is displayed on the display means, when a tap is performed on the touch panel on a portion other than the input field in the composite image, until performing control to replace the composite image with the reflected image, performs control to restrict inputs other than the input of the input information by the input means. The information processing system according to Configuration 7. (Configuration 12) The image is provided with an input field for inputting the input information, The server-side transmission means transmits information regarding the position and size of the input field in the image to the information processing apparatus. The information processing system according to any one of Configurations 1 to 11. (Configuration 13) The input means is a software keyboard displayed on the display means. The information processing system according to any one of Configurations 1 to 12. (Configuration 14) The display means displays an image of a website as the image. The information processing system according to any one of Configurations 1 to 13. (Configuration 15) The information processing apparatus is an image forming apparatus having at least one of a print function, a scan function, and a fax function. The information processing system according to any one of Configurations 1 to 14. (Method 1) A method for controlling an information processing system including an information processing apparatus and a server communicably connected to the information processing apparatus, The steps executed by the information processing apparatus include A display step of displaying image data from the server as an image, An input step of enabling input of input information for the image displayed in the display step, An apparatus-side generation step of generating a composite image by synthesizing the image in the state before the input information is input and the input information input in the input step for the image, An apparatus-side transmission step of transmitting the input information to the server, A control step for controlling the display of the composite image in the display step is included. The steps executed on the server include: A server-side receiving step for receiving the input information from the device-side transmission step; A server-side generating step for generating image data of a reflected image in which the image is reflected in a state where the input information is input, based on the input information received in the server-side receiving step; A server-side transmission step for transmitting the image data of the reflected image to the information processing device is included. The composite image is an image imitating the reflected image. In the control step, after the composite image is displayed in the display step, control is performed to replace the composite image with the reflected image. A control method for an information processing system characterized by this. (Program 1) A program for causing a computer to execute each means of the information processing system according to any one of Configurations 1 to 15.

Explanation of Signs

[0146] 10 Image forming apparatus 100 Image generation system 109 Storage 250 Browser processing unit 300 Controller unit 310 Network I / F 312 Operation unit 352 UI control unit 360 Browser control unit 1000 Cloud browser system

Claims

1. An information processing system including an information processing device and a server communicably connected to the information processing device, The information processing device includes: A display means for displaying the image data from the server as an image; an input means capable of inputting input information for the image displayed on the display means; a device-side generating means for generating a composite image by combining the image in a state before the input information is input with the input information inputted by the input means; a device side transmitting means for transmitting the input information to the server; a control means for controlling the display means to display the composite image, The server, a server-side receiving means for receiving the input information from the device-side transmitting means; a server-side generating means for generating image data of a reflected image by reflecting the image in a state in which the input information was input based on the input information received by the server-side receiving means; a server-side transmitting means for transmitting image data of the reflected image to the information processing device; the composite image is an image that mimics the reflected image, The information processing system is characterized in that the control means performs control to replace the composite image with the reflection image after the composite image is displayed on the display means.

2. The input information includes characters, The information processing system according to claim 1, characterized in that, when generating the composite image, the device-side generation means uses a font for the characters that is the same as or similar to the font used in the image data.

3. The information processing system according to claim 2, wherein when it is impossible to use the same font as the font used in the image data, the device-side generation means uses a font approximated to the font used in the image data.

4. When the image displayed on the display means transitions to another image different from the image upon completion of the input of the input information by the input means, the display means displays a message indicating that the other image is in the process of transitioning. The information processing system according to claim 1.

5. The information processing system according to claim 4, wherein the display means displays a part of the composite image together with the message.

6. The information processing system according to claim 4, wherein the display means displays the input information together with the message.

7. An input field for inputting the input information is provided in the image, The device-side generation means generates, as the composite image, an image in a state where the input information is input in the input field, The information processing system according to claim 1, wherein the server-side generation means generates, as the reflected image, an image in a state where the input information is input in the input field.

8. The display means has a touch panel having a touch function, The control means performs control to enable re-input of the input information by the input means when a tap is performed on the touch panel with respect to the input field in the composite image while the composite image is being displayed on the display means. The information processing system according to claim 7.

9. The display means has a touch panel having a touch function, The control means performs control to cause the display means to display the specific content of the input information when a tap on the touch panel is performed on a portion other than the input field in the composite image while the composite image is being displayed on the display means. The information processing system according to claim 7, characterized in that.

10. The control means performs control to cause the display means to display the specific content of the input information prior to the control of replacing the composite image with the reflected image. The information processing system according to claim 9, characterized in that.

11. The display means has a touch panel having a touch function, The control means, when a tap on the touch panel is performed on a portion other than the input field in the composite image while the composite image is being displayed on the display means, performs control to restrict inputs other than the input of the input information by the input means until control to replace the composite image with the reflected image is performed. The information processing system according to claim 7, characterized in that.

12. The image is provided with an input field for inputting the input information, The server-side transmission means transmits information regarding the position and size of the input field in the image to the information processing apparatus. The information processing system according to claim 1, characterized in that.

13. The input means is a software keyboard displayed on the display means. The information processing system according to claim 1, characterized in that.

14. The display means displays an image of a website as the image. The information processing system according to claim 1, characterized in that.

15. The information processing apparatus is an image forming apparatus having at least one of a printing function, a scanning function, and a fax function. The information processing system according to claim 1, characterized in that.

16. A method for controlling an information processing system including an information processing device and a server communicably connected to the information processing device, The steps executed by the information processing device include: A display step of displaying the image data from the server as an image; An input step in which input information can be input with respect to the image displayed in the display step; An apparatus-side generation step of generating a composite image by combining the image in the state before the input information is input and the input information input in the input step with respect to the image; An apparatus-side transmission step of transmitting the input information to the server; A control step of performing control to display the composite image in the display step, and the control step includes: The steps executed by the server include: A server-side reception step of receiving the input information from the apparatus-side transmission step; A server-side generation step of generating image data of a reflected image in which the image is reflected in a state where the input information is input based on the input information received in the server-side reception step; A server-side transmission step of transmitting the image data of the reflected image to the information processing device, and the server-side transmission step includes: The composite image is an image imitating the reflected image; In the control step, after the composite image is displayed in the display step, control is performed to replace the composite image with the reflected image. A method for controlling an information processing system, characterized in that.

17. A program for causing a computer to execute each means of the information processing system according to claim 1.

Images