Information processing system, server, control method, program product, and storage medium

Abstract

The invention relates to an information processing system, a server, a control method, a program product, and a storage medium. Provided is an information processing system for quickly ascertaining the amount of movement of a movement operation for moving a displayed image. An information processing system includes a server and an information processing apparatus including a display device and an operation device. When the operation device accepts a movement operation, operation information on the movement operation is transmitted to the server. The display apparatus displays a first image before a moving operation, and then displays a second image after the moving operation, the second image including a moving image in a state in which the first image is moved and a blank image that becomes blank according to the moving. The server generates a third image based on the operation information, the third image including an image corresponding to the moving image and an image continuously connected to the moving image. The display device displays the third image transmitted from the server after displaying the second image.

Description

Technical Field

[0001] The present invention relates to an information processing system and server supporting a remote desktop, a control method therefor, a storage medium storing a control program therefor, and a computer program product. Background Art

[0002] In a remote desktop, a screen displayed on a display unit changes according to a user's operation, but a time delay in which a communication time delay occurs may be relatively large. In order to suppress the time delay, there is a known remote desktop system in which an image of a range wider than the screen size is stored (cached) and a portion of the image is extracted from a storage device to display the screen during a scrolling operation (for example, see Japanese Patent Laid-Open No. 2016-110335 (JP2016-110335A)).

[0003] However, since the remote desktop system disclosed in JP2016-110335A is configured to store an image wider than the screen size, it is difficult to cope with a screen having a header container that remains on the display even when a scrolling operation is performed. Also, when a scrolling operation is performed, it is difficult to quickly recognize the amount of movement of the operation or to feel that the image has moved smoothly. Summary of the invention

[0004] The present invention provides an information processing system, a server, a control method therefor, a computer program product, and a storage medium storing the computer program, which enables a user to quickly grasp the movement amount of a moving operation and feel that the image has been moved smoothly when performing a moving operation for moving an image.

[0005] Thus, one aspect of the present invention provides an information processing system, which includes an information processing device and a server communicatively connected to the information processing device. The information processing device includes: a display device that displays image data obtained from the server as an image; an operation device that accepts a user's moving operation for moving an image displayed on the display device; a device-side sending unit that is configured to send operation information about the moving amount of the moving operation to the server after the moving operation ends; and a control unit that is configured to control the display device to display the image before the moving operation as a first image and to display a second image when the moving operation is applied to the first image. The second image includes a moving image in a state where the first image is moved by the moving operation and a blank image that becomes blank according to the movement of the first image. The server includes: a server-side receiving unit configured to receive the operation information from the device-side sending unit; a generating unit configured to generate image data of a third image based on the operation information received by the server-side receiving unit, the third image including an image corresponding to the moving image and an image continuously connected to the moving image and filling the blank image; and a server-side sending unit configured to send the image data of the third image to the information processing device. The control unit controls the display device to replace the second image with the third image after displaying the second image.

[0006] According to the present invention, when the user performs a move operation for moving an image, it is possible to quickly grasp the movement amount of the move operation and feel that the image has been moved smoothly.

[0007] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram showing the entire configuration of the cloud browser system according to the first embodiment.

[0009] Figure 2A is a block diagram illustrating an example of a hardware configuration of a virtual machine.

[0010] Figure 2B is a block diagram illustrating an example of a software configuration of a virtual machine.

[0011] Figure 3A is a block diagram illustrating an example of a hardware configuration of an image forming apparatus.

[0012] Figure 3Bis a block diagram showing an example of a software configuration of the image forming apparatus.

[0013] Figure 4 is a view showing an example of a home screen displayed on an operation device of the image forming apparatus.

[0014] Figure 5 is a view showing an example of a browser screen displayed on an operation device of the image forming apparatus.

[0015] Fig. 6A is a sequence diagram showing an example of processing performed between the image forming apparatus, the image generating system, and the external site.

[0016] Figure 6B is a sequence diagram showing an example of processing performed between the image forming apparatus, the image generating system, and the external site.

[0017] Figure 6C is a flowchart showing a process (drag operation determination process) executed in the image forming apparatus.

[0018] Figure 7 is a sequence diagram showing an example of processing performed between the image forming apparatus, the image generating system, and the external site.

[0019] FIG. 8A to FIG. 8G is a view showing an example of an image displayed on an operating device of the image forming apparatus.

[0020] 9A to 9C is a view showing an example of an image displayed on an operating device of the image forming apparatus.

[0021] Fig.9D and Fig.9E A view used to describe the start and end points of an image that is moved by dragging.

[0022] Fig. 10A and Fig. 10B is a flowchart showing processing executed by the image forming device (browser controller).

[0023] Fig.11A and Fig. 11B is a sequence diagram showing an example of processing performed among an image forming apparatus, an image generating system, and an external site in a cloud browser system according to the second embodiment.

[0024] FIG. 12A to FIG. 12F is a view showing an example of an image displayed on an operating device of the image forming apparatus.

[0025] Fig.13A and Fig. 13Bis a view showing an example of frame information related to a frame image.

[0026] Fig.14 is a flowchart showing the processing performed by the image generation system (browser processor).

[0027] Fig.15 is a flowchart showing processing executed by the image forming device (browser controller).

[0028] FIG. 16A to FIG. 16D : is a view showing an example of an image displayed on an operating device of an image forming apparatus of a cloud browser system according to the third embodiment.

[0029] Fig.17 is a sequence diagram showing an example of processing performed between the image forming apparatus, the image generating system, and the external site.

[0030] Fig.18 is a flowchart illustrating processing performed by the image generation system.

[0031] Fig.19 is a diagram showing the image generation system in relation to Fig.18 Flowchart of processing executed at a timing different from the execution timing of the flowchart shown in .

[0032] Fig. 20 : is a flowchart showing the processing performed by the image generation system of the cloud browser system according to the fourth embodiment. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments. For example, each of the parts constituting the present invention can be replaced with any configuration that can present the same function. In addition, any component can be added. In addition, two or more arbitrary configurations (features) of the embodiments can be combined.

[0034] Now refer to Figures 1 to 10B A first embodiment is described. Figure 1 is a block diagram showing the entire configuration of the cloud browser system according to the first embodiment. Figure 1The cloud browser system 1000 shown in is an information processing system capable of processing various information. The cloud browser system 1000 includes a plurality of image forming devices 101, 102, and 103 as information processing devices and an image generation system 100, which are communicably connected to each other. The cloud browser system 1000 is communicably connected to an external site 105 and an authentication site 110 via the Internet 111. Each of the image forming devices 101, 102, and 103 is connected to the Internet 111 via a proxy server 104.

[0035] In this embodiment, the cloud browser system 1000 has three image forming devices 101, 102, and 103, but the number of image forming devices arranged is not limited thereto, and may be, for example, one, two, or four or more. In the following, the image forming device 101 is represented among the image forming devices 101, 102, and 103. The cloud browser system 1000 includes an image generating system 100, but the number of image generating systems 100 to be arranged is not limited thereto, and may be, for example, two or more. The number of image generating systems 100 to be arranged is preferably less than the number of image forming devices to be arranged. In the cloud browser system 1000, the proxy server 104 may be omitted. For example, a user using the image forming device 101 will view a screen displayed on the image forming device 101. In this case, the user operates the image forming device 101 to input a request for browsing the external site 105 to be browsed. In response to this, the image forming device 101 sends the address (URL) of the external site 105 to the image generating system 100 via the proxy server 104. The external site 105 is an http server that returns an http response to an http (Hypertext Transfer Protocol) request, but its configuration is not particularly limited.

[0036] The image generation system 100 is a server having a gateway 106, a virtual machine 107, and a storage device 109. A browser engine as a software module operates on the virtual machine 107. The browser engine receives the URL of the external site 105 sent from the image forming device 101 via the gateway 106. Then, the browser engine accesses the external site 105 corresponding to the URL via the gateway 106, and receives Web content such as HTML from the external site 105. Thereafter, a rendering result of the Web content is generated by a software module that performs rendering. The rendering result is sent to the image forming device 101 via the gateway 106. The image forming device 101 receives the rendering result sent from the image generation system 100 and displays the rendering result. This allows the user to check the content of the external site 105. Here, "rendering" generally means displaying a browser screen (an image of a website supplied from the external site 105) based on the Web content. In this embodiment, "rendering" indicates that the Web content such as HTML received from the external site 105 is converted into image data of a predetermined screen size.

[0037] The storage device 109 stores user data such as cookies obtained by interpreting web content received from the external site 105 by the browser engine operating on the virtual machine 107. The cloud browser system 1000 is configured to be communicable with the authentication site 110. Users using the image forming devices 101, 102, and 103 can be identified by the authentication site 110. The image generation system 100 obtains the authentication of the user based on the user information sent from any one of the image forming devices 101, 102, and 103. The image generation system 100 manages the user data stored in the storage device 109 for each authenticated user. In this embodiment, the image generation system 100 includes a virtual machine 107, but the number of virtual machines to be arranged is not limited thereto, and multiple virtual machines may be included. In this case, it is preferred to arrange a load balancer between the gateway 106 and each virtual machine 107. In addition, it is preferred to select a virtual machine 107 from among the multiple virtual machines 107 according to the load state. The storage device 109 is preferably used together with the multiple virtual machines 107.

[0038] Figure 2A is a block diagram showing an example of a hardware configuration of a virtual machine. Figure 2AAs shown in , the virtual machine 107 includes a CPU 201, an internal storage device 202, a RAM 203, an interface 204, and a communication interface 205 as hardware components, and these are connected via a bus 206. The CPU 201 performs various processes using computer programs and data stored in the internal storage device 202. The computer program includes, for example, a program for causing a computer to execute each unit or each component in the cloud browser system 1000 (a control method for an information processing system and a control method for a server). This allows the CPU 201 to control the operation of the entire virtual machine 107. The internal storage device 202 stores setting data of the virtual machine 107, computer programs and data related to the activation of the virtual machine 107, computer programs and data related to the basic operation of the virtual machine 107, and the like. The RAM 203 has an area for storing computer programs and data loaded from the internal storage device 202, data received via the communication interface 205, and the like. The RAM 203 also has a work area used when the CPU 201 performs various processes. In this way, the RAM 203 can provide various areas (storage areas) as appropriate. The interface 204 includes a display unit for displaying the result of the processing performed by the CPU 201 using images, characters, etc., and an operating device operated by the user for performing various operation inputs. The display unit includes, for example, a liquid crystal screen and a touch panel screen. The operating device includes, for example, a keyboard, a mouse, a touch panel screen, etc. The communication interface 205 is an interface for performing data communication with an external device. Specifically, the communication interface 205 can receive image data from the image forming devices 101 to 103 (server-side receiving step), and can send image data to the image forming devices 101, 102 and 103 (server-side sending step). As described above, the communication interface 205 in this embodiment has a function as a receiving unit (server-side receiving unit) for receiving image data and a function as a sending unit (server-side sending unit) for sending image data. In the virtual machine 107, a part used as a receiving unit and a part used as a sending unit can be provided separately.

[0039] The hardware configuration of the virtual machine 107 is not limited to Figure 2A , and for example, a memory device may also be connected to the bus 206. The memory device is not particularly limited, and examples thereof include a hard disk drive, a USB memory, a magnetic card, an optical card, an IC card, a memory card, and a drive device for a floppy disk (FD) or an optical disk such as a compact disk (CD). The virtual machine 107 may be configured by virtualization technology, and may organize various resources constituting the computer system into logical units independently of the physical configuration.

[0040] Figure 2B is a block diagram showing an example of a software configuration of a virtual machine. Figure 2BAs shown in , the virtual machine 107 includes a browser processing module 250 and a network control module 254 as software. The browser processing module 250 includes an overall control module 251, a browser engine 252, and a rendering control module 253. The browser processing module 250 is a module that performs overall processing related to the browser performed on the image generation system 100 side (which is the cloud). The overall control module 251 is a module that controls the entire browser processing unit 250 operating on the virtual machine 107. When the user operates the image forming device 101 to input a browsing request for the external site 105 that the user wants to browse, the URL of the external site 105 is sent from the image forming device 101 to the image generation system 100 via the proxy server 104.

[0041] The overall control module 251 collectively receives various requests sent from the image forming apparatus 101. The overall control module 251 appropriately distributes the requests received from the image forming apparatus 101 to the browser engine 252 and the rendering control module 253. The browser engine 252 accesses the external site 105 via the network control module 254 based on the URL sent from the overall control module 251. Therefore, the web content of the external site 105 can be obtained. The browser engine 252 includes the function of the http client required to obtain the web content. The browser engine 252 can analyze the web content, appropriately store information such as cookies in the storage device 109, and refer to the information such as cookies pre-stored in the storage device 109. When the analysis of the web content is completed, the browser engine 252 requests the rendering control module 253 to perform rendering processing. The rendering control module 253 renders the web content based on the rendering request from the browser engine 252. The rendering control module 253 sends the rendering result to the image forming apparatus 101 as the request source of the URL via the network control module 254. The network control module 254 is a module that relays communication between the browser processing module 250 and an external device such as the external site 105 or the authentication site 110 by controlling the communication interface 205. The network control module 254 includes a protocol stack for http communication required to access the external site 105.

[0042] As described above, in the cloud browser system 1000, the user can input a browsing request for the external site 105 that the user wants to browse by operating the image forming device 101. As a result, the URL of the external site 105 is sent from the image forming device 101 to the image generation system 100 via the proxy server 104. The overall control module 251 receives the URL of the external site 105 and transmits it to the browser engine 252. This allows the browser engine 252 to receive the URL. Then, the browser engine 252 requests the network control module 254 to access the external site 105 indicated by the URL in order to obtain the Web content indicated by the URL. When receiving the access request, the network control module 254 controls the communication interface 205 to access the external site 105 via the gateway 106 and the Internet 111. The network control module 254 obtains the Web content from the external site 105 and transmits it to the browser engine 252.

[0043] The browser engine 252 sends and receives user data such as cookies to and from the storage device 109 as needed based on the Web content transmitted from the network control module 254, and analyzes the Web content. The browser engine 252 requests the rendering control module 253 to render the Web content according to the analysis of the Web content. When the request is received, the rendering control module 253 performs the rendering of the Web content. When the rendering is completed, the rendering control module 253 notifies the completion to the overall control module 251. The overall control module 251 notifies the image forming device 101 of the URL request source of the completion of the rendering via the network control module 254. When the completion of the rendering is received, the image forming device 101 requests to obtain the rendering result from the image generation system 100. This request is transmitted to the overall control module 251 via the network control module 254 and transmitted to the rendering control module 253 by the overall control module 251. Then, when a request to obtain the rendering result is received, the rendering control module 253 sends the rendering result to the image forming device 101 as the request source via the network control module 254. This allows the image forming device 101 to display the rendering result. The user can browse the contents of a desired external site 105 on the image forming apparatus 101 .

[0044] Figure 3A 1 is a block diagram showing an example of the hardware configuration of the image forming apparatus 101. The image forming apparatuses 102 and 103 also have the same configuration. Figure 3AAs shown in FIG. 1 , the image forming apparatus 101 includes a controller unit 300 (controller), an operating device 312, a scanner 370, and a printer 395 as hardware. The controller unit 300 includes a CPU 301, a RAM 302, a ROM 303, a storage device (storage unit) 304, an image bus I / F 305, an operating unit I / F 306, a network I / F 310, a USB host I / F 313, and an RTC 315. These are communicably connected to each other via a system bus 307. The controller unit 300 includes a device I / F 320, a scanner image processor 380, and a printer image processor 390. These are communicably connected to each other via an image bus 308. The system bus 307 and the image bus 308 are connected to each other via the image bus I / F 305.

[0045] The controller unit 300 controls the operation of the operating device 312, the scanner 370, and the printer 395 (control unit). For example, the controller unit 300 realizes a printing function (copying function) by controlling the scanner 370 as an image input device to read image data and controlling the printer 395 as an image output device to print out the image data. The CPU 301 starts the operating system (OS) 351 (see the operating system 351) through the boot program stored in the ROM 303. Figure 3B ). The CPU 301 executes the program stored in the storage device 304 on the OS. The RAM 302 is used as a work area of ​​the CPU 301. The RAM 302 provides a work area and an image memory area for temporarily storing image data. For example, the program and the image data are stored in the data storage area (storage unit) 108 of the storage device 304. The program is not particularly limited, and, for example, a control program that causes the computer to execute each unit or each component of the cloud browser system 1000 (a control method for an information processing system) is illustrated. The control program may be stored in the internal storage device 202 of the image generating system 100, or may be distributed and stored in the storage device 304 of the image forming apparatus 101 and the internal storage device 202 of the image generating system 100. The image data is not particularly limited, and, for example, an image from an external site 105 obtained via the image generating system 100 is illustrated.

[0046] In this embodiment, the operation device 312 is configured by a touch panel having a touch function. Therefore, the operation device 312 can display the image data from the image generation system 100 as an image (first, second or third image display step), for example. The touch panel can be a pressure-sensitive touch panel or an electrostatic touch panel. In the operation device 312, the user can perform a moving operation (operation receiving step) of moving the image displayed on the touch panel by dragging (scrolling) on ​​the touch panel, that is, sliding a fingertip on the touch panel. As described above, the operation device 312 in this embodiment has a function as a display unit for displaying an image and a function as an operation device capable of realizing an image moving operation by the touch function of the touch panel. In the image forming apparatus 101, a part having a function as a display unit and a part having a function as an operation device can be provided separately. The operation unit I / F 306 is an interface responsible for connection with the operation device 312, and outputs the image data to the operation device 312. Therefore, the operation device 312 can display an image of the image data. The operation unit I / F 306 sends input information (operation information) input from the user via the operation device 312 to the CPU 301.

[0047] The network I / F 310 is an interface for connecting the image forming apparatus 101 to a LAN. Thus, the network I / F 310 can send, for example, input information to the image generating system 100 (device-side sending step). In addition, the network I / F 310 can send image data obtained by the scanner 370 to an external device. When the sending of the image data is completed, the network I / F 310 can notify the image generating system 100 of the sending completion information to that effect. The network I / F 310 can also receive various information from the image generating system 100. In this way, the network I / F 310 in this embodiment has a function as a receiving unit (device-side receiving unit) that receives various information and a function as a sending unit (device-side sending unit) that sends various information. In the image forming apparatus 101, a portion used as a receiving unit and a portion used as a sending unit can be provided separately. The USB host I / F 313 is an interface for communicating with the USB storage device 314. The USB host I / F 313 can send data stored in the USB storage device 314 to the CPU 301. The USB host I / F 313 also functions as an output unit that stores data stored in the storage device 304 in the USB storage device 314. The USB storage device 314 is an external storage device that stores data and is detachable from the USB host I / F 313. A plurality of USB storage devices 314 can be connected to the USB host I / F 313. The RTC 315 counts the current time. The time information counted by the RTC 315 is used to record the job input time, etc.

[0048] The image bus I / F 305 is a bus bridge for connecting the system bus 307 and the image bus 308 that transmits image data at a high speed and converting the data format. The image bus 308 is configured by a PCI bus or the like. The scanner 370 and the printer 395 are connected to the device I / F 320. The device I / F 320 performs conversion between the synchronous and asynchronous systems of the image data. The scanner image processor 380 corrects, processes or edits the input image data, for example. The printer image processor 390 corrects the print output image data or converts its resolution depending on the printer 395. As described above, in this embodiment, the image forming apparatus 101 is an apparatus having a printing function and a scanning function, but this is not restrictive, and the image forming apparatus 101 may be an apparatus having at least one of a printing function, a scanning function, and a fax function.

[0049] Figure 3B 1 is a block diagram showing an example of the software configuration of the image forming apparatus 101. Figure 3B As shown in FIG. 1 , the image forming apparatus 101 includes a browser control module 360 ​​and a UI control module 352, a job control module 353, a network control module 354, and a storage control module 355 as software. The browser control module 360 ​​includes an image data obtaining module 361, a browser operation module 362, a browser display module 363, a command I / F module 364, and a proxy processing module 365. Figure 3B Each module surrounded by a solid line in is a software module implemented by executing a main program loaded into the RAM 302 by the CPU 301. The main program manages and controls the execution of each module by using the OS 351. The UI control module 352 displays an image (screen) on the operating device 312, and receives an operation from a user on the operating device 312 via the operating unit I / F 306. The UI control module 352 can also control screen updates by communicating with another module and receiving a rendering instruction from another module. The job control module 353 is a module that receives a job execution instruction from the UI control module 352 and controls job processing such as copying, scanning, and printing. In this embodiment, when an operation of selecting a file is requested via the browser operation module 362, the browser operation module 362 instructs the job control module 353 to execute a scan job via the UI control module 352. The network control module 354 receives a communication request from another module and controls the network I / F 310, thereby controlling communication with an external device. The network control module 354 can also receive a notification from an external device and notify another module of the content of the notification. The storage control module 355 records and manages setting information and job information recorded in the storage device 304. Each module located in the layer of the OS 351 accesses the storage control module 355 to refer to and set setting values.

[0050] The browser control module 360 ​​is a submodule included in the OS 351. When receiving a notification of a user operation from the UI control module 352, the browser operation module 362 notifies the command I / F module 364 or the proxy processing module 365 of the content of the user operation. When receiving the notification from the browser operation module 362, the proxy processing module 365 requests to obtain proxy setting information from the storage control module 355. When determining that the proxy setting is valid based on the proxy setting information obtained from the storage control module 355, the proxy processing module 365 sends a communication request to the proxy server 104 via the network control module 354. The proxy processing module 365 receives a response to the communication request via the network control module 354. The proxy processing unit 365 notifies the browser display module 363 or the command I / F module 364 of the result of processing the content of the response.

[0051] When receiving notifications from the browser operation module 362 and the proxy processing module 365, the command I / F module 364 requests communication with the image generation system 100 via the network control module 354. The communication request at this time may include information notified from the browser operation module 362 and the proxy processing module 365. This information also includes information about user operations (such as text input, button pressing, dragging and scaling). Information about text input includes, for example, a URL. Information about button pressing includes the coordinates of the pressed position on the operating device 312. Information about dragging and scaling includes a string associated with each operation. The command I / F module 364 receives communication from the image generation system 100 via the network control module 354. The command I / F module 364 processes the communication content from the image generation system 100 and notifies the image data acquisition module 361 or the browser display module 363 of the communication content. The image data acquisition module 361 receives a completion notification of the rendering result from the command I / F module 364. When receiving the completion notification, the image data acquisition module 361 requests to obtain an image from the image generation system 100. The image acquisition request is sent to the rendering control module 253 operating on the virtual machine 107 of the image generation system 100. The image is then sent out and the rendering result is sent to the image data acquisition module 361. When receiving the rendering result, the image data acquisition module 361 transmits the image as the rendering result to the browser display module 363. The browser display module 363 receives the image from the image data acquisition module 361 and instructs the UI control module 352 to draw the image. In addition, the browser display module 363 receives notifications from the command I / F module 364 and the proxy processing module 365, and instructs the UI control module 352 to display a message corresponding to the notification.

[0052] Figure 4 is a view showing an example of a home screen displayed on an operation device of the image forming apparatus. Figure 4The main screen 400 shown in includes buttons 401, 402, 403 and 404. By operating, that is, pressing button 401, for example, sending an email to cloud A (not shown) is enabled. By operating button 402, for example, sending an email to cloud B (not shown) is enabled. By operating button 403, the browser control module 360 ​​is activated. By operating button 404, the printer 395 is set to a state that enables standard size printing. As described above, the main screen 400 includes four buttons for the user to indicate the functions that can be executed by the image forming device 101. The number of buttons included in the main screen 400 is not limited to four and can be changed appropriately.

[0053] Figure 5 is a view showing an example of a browser screen displayed on an operation device of the image forming apparatus. Figure 5 The browser screen 500 shown in FIG. 4 is displayed when the browser control module 360 ​​is activated by the operation of the button 403 on the main screen 400. The browser screen 500 includes a back button 501, a forward button 502, an address bar 503, a setting button 504, and a content area 505. By operating the HOME key (not shown) provided in the operating device 312, the browser control module 360 ​​in the activated state can be forcibly terminated. The URL "https: / / **.***.***.*** / " is displayed in the address bar 503. This URL is the address of the external site 105. In the content area 505, the result of rendering the web content obtained from the external site 105, that is, the screen of the website is displayed. Note that after the browser control module 360 ​​is activated, the rendering result, etc. are not displayed in the content area 505 until the URL is entered into the address bar 503. Furthermore, a default URL may be registered in an item included in the setting button 504 in the browser screen 500 , and a result of rendering the Web content of the relevant URL may be displayed in the content area 505 simultaneously with the activation of the browser control module 360 ​​.

[0054] Fig. 6A 1 is a sequence diagram showing an example of processing performed between an image forming apparatus, an image generating system, and an external site. Fig. 6A As shown in FIG. 1 , in step S601, the user of the image forming apparatus 101 operates (presses) the main screen 400 (see FIG. 100 ) displayed on the operation device 312 of the image forming apparatus 101. Figure 4 ) on button 403.

[0055] In step S602, the UI control module 352 of the image forming apparatus 101 sends an activation request to the browser control module 360. As a result, the browser control module 360 ​​is activated, and the browser screen 500 is displayed on the operation device 312 of the image forming apparatus 101 (see Figure 5 ).

[0056] In step S603a, the browser control module 360 ​​of the image forming apparatus 101 sends a rendering request for rendering a web page to the browser processing module 250 (virtual machine 107) of the image generating system 100. The rendering request includes, for example, URL information indicating the location of the web content to be displayed on the browser screen 500, and device information of the image forming apparatus 101. The "URL information" is information about a URL that is pre-set as a setting value of the button 403 or information about a URL input in the address bar 503 on the browser screen 500. The "device information" is, for example, screen size information of the operating device 312 of the image forming apparatus 101 and screen type information for specifying the operating device 312, but is not limited thereto.

[0057] In step S603b, the browser control module 360 ​​of the image forming apparatus 101 requests the UI control module 352 to display a loading screen on the operation device 312, and the UI control module 352 renders the loading screen. The loading screen is a screen for notifying the user that processing is in progress until the browser screen 500 is displayed. The notification screen is not limited to the loading screen, and may be, for example, a pop-up screen.

[0058] When receiving the rendering request in step S603a, the browser processing module 250 checks the presence or absence of the user agent information stored in the storage device 109 based on the device information in the rendering request in step S604.

[0059] When the user agent information exists in the storage device 109 , the browser processing module 250 obtains the user agent information from the storage device 109 in step S605 .

[0060] In step S606 , the browser processing module 250 requests to obtain Web content from the external site 105 based on the URL information obtained in response to the rendering request in step S603 a and the user agent information obtained in step S605 .

[0061] In step S607 , the browser processing module 250 receives the Web content from the external site 105 in response to the request to obtain the Web content in step S606 .

[0062] In step S608 , the browser processing module 250 renders the web content received in step S607 .

[0063] In step S609 a , the browser processing module 250 stores the result of the rendering performed in step S608 in the storage device 109 .

[0064] In step S609 b , the browser processing module 250 notifies the browser control module 360 ​​included in the image forming apparatus 101 of URL information indicating the location of the image data of the rendering result stored in step S609 a .

[0065] In step S610, the browser control module 360 ​​of the image forming apparatus 101 requests acquisition of image data from the storage device 109 of the image generating system 100 based on the URL information notified in step S609b. This request is performed using the URL information notified in step S609b.

[0066] In step S611 , the browser control module 360 ​​receives the image data from the storage device 109 as a result of the request to obtain the image data in step S610 .

[0067] In step S612, the browser control module 360 ​​instructs the UI control module 352 to display the image data received in step S611 in the content area 505 on the browser screen 500. As a result, the browser screen 500 including the image of the content area 505 of the size suitable for the operation device 312 (i.e., the screen size of the touch panel) is displayed on the operation device 312 of the image forming apparatus 101.

[0068] Figure 6B 1 is a sequence diagram showing an example of processing performed between an image forming apparatus, an image generating system, and an external site. Figure 6B As shown in FIG. , in step S641, Fig. 6A As in the process of step S606 in FIG. 1 , the browser processing module 250 of the image generating system 100 requests to obtain the Web content from the external site 105 .

[0069] In step S642, Fig. 6A As with the processing in step S607 in , the browser processing module 250 receives web content from the external site 105 in response to the request to obtain web content in step S641.

[0070] In step S643, Fig. 6A The browser processing module 250 renders the web content, just like the process in step S608 in FIG.

[0071] In step S644a, Fig. 6ASimilar to the processing in step S609a in , the browser processing module 250 stores (saves) the result of the rendering performed in step S643 in the storage device 109.

[0072] In step S644b, Fig. 6A As in the process in step S609b in , the browser processing module 250 notifies the browser control module 360 ​​included in the image forming apparatus 101 of URL information indicating the location of the image data of the rendering result stored in step S644a.

[0073] In step S645, Fig. 6A As in the process in step S610 in , the browser control module 360 ​​of the image forming apparatus 101 requests to obtain image data from the storage device 109 of the image generating system 100 .

[0074] In step S646, Fig. 6A As with the processing in step S611 in , the browser control module 360 ​​receives the image data from the storage device 109 as a result of the image data acquisition request in step S645.

[0075] In step S647, the browser control module 360 ​​instructs the UI control module 352 to render the image data received in step S646. Thus, the image of the image data is displayed (rendered) on the operating device 312.

[0076] In step S648, the user performs a tap-in operation on the image displayed on the operation device 312 in step S647. The "tap-in" is an operation in which the user touches or presses the touch panel with a fingertip or the like. Then, the operation device 312 sends an electrical signal corresponding to the position of the fingertip (tap-in) on the touch panel to the UI control module 352 (CPU 301). Therefore, the UI control module 352 can specify the coordinates of the position of the fingertip based on the electrical signal.

[0077] In step S649 , the UI control module 352 of the image forming apparatus 101 transmits information on the coordinates specified in step S648 to the browser control module 360 ​​as a tap enter event.

[0078] In step S648, the user can also perform dragging by sliding the fingertip on the touch panel, for example, upward, downward, rightward, or leftward after tapping into. In this case, the electrical signal also changes according to the change in the position of the fingertip on the touch panel. Therefore, the UI control module 352 can specify the coordinates that change according to the movement of the fingertip based on the electrical signal at each predetermined time interval. Then, in step S649, as a drag event, the UI control module 352 sends information about the coordinates that change according to the movement of the fingertip and information about the movement amount of the fingertip to the browser control module 360.

[0079] In step S650 , the browser control module 360 ​​stores the information sent from the UI control module 352 in step S649 in the RAM 302 .

[0080] In step S651, the user performs a tap-out (tap-out operation) on the image displayed on the operation device 312 in step S647. "Tap-out" refers to an operation of releasing a fingertip from a position where the user touches the touch panel by tapping in or dragging. Then, an electrical signal corresponding to the position where the fingertip is separated from the touch panel (tap-out) is sent from the operation device 312 to the UI control module 352. Therefore, the UI control module 352 can specify the coordinates of the position where the fingertip is separated based on the electrical signal.

[0081] In step S652, as a tap-to-leave event, the UI control module 352 of the image forming device 101 sends information about the coordinates identified in step S651 to the browser control module 360. Note that the operation of performing a tap-to-enter, then sliding the fingertip on the touch panel faster than a normal drag, and then performing a touch-to-leave is called a "flick". In addition, the swipe operation can be divided into dragging and tap-to-leave. In this case, the processing when dragging and the processing when tapping-to-leave are performed. This enables image movement (scrolling) by swiping. As with the tap-to-leave event and the drag event, as a swipe event, the UI control module 352 can send information about the coordinates of the swipe to the browser control module 360.

[0082] The browser control module 360 ​​determines whether the operation performed by the user in steps S648 to S651 is an operation including dragging (swiping). This determination will refer to Figure 6C Give a description.

[0083] Figure 6C 2 is a flowchart showing a process (drag operation determination process) executed in the image forming apparatus. Figure 6CAs shown in , in step S691, the browser control module 360 ​​determines whether dragging or swiping has been performed (detected). This determination is performed based on whether the coordinates of the tap-enter event sent in step S649 are the same as the coordinates of the tap-leave event sent in step S652. Specifically, when the coordinates of the two are different from each other, it is determined that dragging has been performed, and when the coordinates of the two are the same as each other, it is determined that dragging has not been performed. Then, as a result of the determination in step S691, when it is determined that dragging is performed, the processing proceeds to step S692. On the other hand, if it is determined in step S691 that dragging is not performed, then the processing proceeds to step S693.

[0084] In step S692, the browser control module 360 ​​determines that the operation performed by the user in steps S648 to S651 is a drag operation.

[0085] In step S693, the browser control module 360 ​​determines that the operation performed by the user in steps S648 to S651 is a tap operation, that is, a simple pressing operation. Note that here, as an example, it is assumed that the operation is determined to be a tap operation. The case where the operation includes dragging will be referred to later. Figure 7 Give a description.

[0086] Then, in step S653 , the browser control module 360 ​​transmits the start coordinates of the tap operation (ie, information on the coordinates at the time of tap entry) to the browser processing module 250 of the image generating system 100 as operation information.

[0087] In step S654, the browser processing module 250 analyzes the presence or absence of the URL of the link destination at the coordinates, the presence or absence of the input field of the text, etc. based on the coordinate information sent in step S653. In this example, as a result of the analysis, it is assumed that the URL of the link destination exists.

[0088] In step S655 , the browser processing module 250 requests to obtain the URL of the link destination as the analysis result in step S654 , that is, the Web content, from the external site 105 .

[0089] In step S656 , the browser processing module 250 receives the web content from the external site 105 in response to the request to obtain the web content in step S655 .

[0090] In step S657 , the browser processing module 250 renders the web content received in step S656 .

[0091] In step S658 a , the browser processing module 250 stores the result of the rendering performed in step S657 in the storage device 109 .

[0092] In step S658 b , the browser processing module 250 notifies the browser control module 360 ​​included in the image forming apparatus 101 of URL information indicating the location of the image data of the rendering result stored in step S658 a .

[0093] In step S659 , the browser control module 360 ​​of the image forming apparatus 101 requests to obtain image data from the storage device 109 of the image generating system 100 based on the URL information notified in step S658 b .

[0094] In step S660 , the browser control module 360 ​​receives the image data from the storage device 109 as a result of the request to obtain the image data in step S659 .

[0095] In step S661, the browser control module 360 ​​instructs the UI control module 352 to render the image data received in step S660. As a result, the image of the link destination analyzed in step S654 is displayed on the operation device 312.

[0096] Figure 7 1 is a sequence diagram showing an example of processing performed between an image forming apparatus, an image generating system, and an external site. Figure 7 As shown in FIG. , in step S701, Fig. 6A As in the process of step S606 in FIG. 1 , the browser processing module 250 of the image generating system 100 requests to obtain the Web content from the external site 105 .

[0097] In step S702, Fig. 6A As with the processing in step S607 in FIG. 1 , the browser processing module 250 receives web contents from the external site 105 as a response to the web contents acquisition request in step S701.

[0098] In step S703, Fig. 6A The browser processing module 250 renders the web content, just like the process in step S608 in FIG.

[0099] In step S704a, Fig. 6A Similar to the processing in step S609a in , the browser processing module 250 stores the result of the rendering performed in step S703 in the storage device 109.

[0100] In step S704b, Fig. 6A As in the process in step S609b in , the browser processing module 250 notifies the browser control module 360 ​​included in the image forming apparatus 101 of URL information indicating the location of the image data of the rendering result stored in step S704a.

[0101] In step S705, Fig. 6A As in the process in step S610 in , the browser control module 360 ​​of the image forming apparatus 101 requests to obtain image data from the storage device 109 of the image generating system 100 .

[0102] In step S706, Fig. 6A As with the processing in step S611 in , the browser control module 360 ​​receives the image data from the storage device 109 as a result of the image data acquisition request in step S705.

[0103] In step S707, the browser control module 360 ​​instructs the UI control module 352 to render the image data received in step S706. As a result, an image of the image data is displayed on the operating device 312. For example, Fig. 8A , Fig.8D or Fig. 9A Each image will be described later.

[0104] In step S708, the user performs a tap-enter operation on the image displayed in step S707 on the operation device 312. Then, an electrical signal corresponding to the tap-enter is transmitted from the operation device 312 to the UI control module 352. Therefore, the UI control module 352 can specify the coordinates of the position of the fingertip based on the electrical signal.

[0105] In step S709 , the UI control module 352 of the image forming apparatus 101 transmits information on the coordinates specified in step S708 to the browser control module 360 ​​as a tap enter event.

[0106] In step S710 , the browser control module 360 ​​stores the information (coordinates of the tap enter position) transmitted from the UI control module 352 in step S709 in the RAM 302 .

[0107] In step S711, the user performs dragging by sliding the fingertip that tapped into the image in step S708 on the touch panel as it is. Then, an electrical signal that changes according to the dragging is sent from the operation device 312 to the UI control module 352. Therefore, the UI control module 352 can specify the coordinates that change according to the movement of the fingertip based on the electrical signal.

[0108] In step S712, the UI control module 352 transmits information on the coordinates specified in step S711 and changing with the movement of the fingertip to the browser control module 360 ​​as a drag event.

[0109] In step S713, the browser control module 360 ​​calculates the coordinates of the start point and the end point in the drag operation based on the information sent from the UI control module 352 in step S712. The coordinates of the start point are the same as the coordinates of the tap-in position in step S708. The calculation results of the browser control module 360 ​​are stored in the RAM 302.

[0110] In step S714, the browser control module 360 ​​instructs the UI control module 352 to render the image data based on the coordinates of the start point and the coordinates of the end point calculated in step S713. Thus, a partial image of the image data is displayed on the operating device 312. For example, here is displayed Figure 8B , Fig. 8E , Figure 8F or Fig. 9B A partial image is shown in . Each image will be described later.

[0111] In step S715, the user performs a tap-off to release the fingertip from the image dragged in step S711. Then, an electrical signal corresponding to the position of the tap-off is sent from the operation device 312 to the UI control module 352. Therefore, the UI control module 352 can specify the coordinates of the position where the fingertip is separated based on the electrical signal. The coordinates of the position where the fingertip is separated are the same as the coordinates of the end point calculated in step S713.

[0112] In step S716 , the UI control module 352 of the image forming apparatus 101 transmits information on the coordinates specified in step S715 to the browser control module 360 ​​as a tap-away event.

[0113] Next, the browser control module 360 ​​determines whether the operation performed by the user in steps S708 to S715 is an operation including dragging. Note that here, as an example, it is assumed that the operation is determined to include dragging. Then, the browser control module 360 ​​calculates the amount of movement of the fingertip during dragging (the amount of scrolling of the image) in step S717 based on, for example, the coordinates of the starting point and the coordinates of the ending point calculated in step S713. Specifically, the browser control module 360 ​​calculates the distance between the coordinates of the starting point and the coordinates of the ending point as the amount of movement of the fingertip. In step S717, the browser control module 360 ​​can calculate the difference between the coordinates of the tap entry position and the coordinates of the tap exit or swipe position stored in step S710, and determine whether the difference is greater than a predetermined threshold. Therefore, when the amount of movement caused by dragging is small, the scrolling of the image can be suppressed.

[0114] In step S718 , the browser control module 360 ​​transmits information on the movement amount calculated in step S717 to the browser processing module 250 of the image generating system 100 .

[0115] In step S719, the browser processing module 250 performs scroll processing. The scroll processing is realized by the browser processing module 250 linking the operation amount (rotation amount) of the mouse wheel to the movement amount of the image. The browser processing module 250 performs processing to generate image data based on the movement amount sent in step S718.

[0116] In step S720 a , the browser processing module 250 stores the image data generated in step S719 in the storage device 109 .

[0117] In step S720 b , the browser processing module 250 notifies the browser control module 360 ​​included in the image forming apparatus 101 of URL information indicating the location of the image data stored in step S720 a .

[0118] In step S721 , the browser control module 360 ​​of the image forming apparatus 101 requests acquisition of image data from the storage device 109 of the image generating system 100 based on the URL information notified in step S720 b .

[0119] In step S722, as a result of the request to obtain image data in step S721, the browser control module 360 ​​receives image data from the storage device 109. The image data is the data stored in step S720a.

[0120] In step S723, the browser control module 360 ​​instructs the UI control module 352 to render the image data received in step S722. As a result, an image of the image data is displayed on the operating device 312. For example, Figure 8C , Figure 8G or Fig. 9C Each image will be described later.

[0121] FIG. 8A to FIG. 8C 31 is a view showing an example of an image displayed on an operating device of an image forming apparatus. As described above, in the image forming apparatus 101, a user can perform a moving operation (hereinafter, referred to as an "image moving operation") of moving an image displayed on the touch panel by dragging on the operating device 312 (on the touch panel). Fig. 8A An image displayed before the move operation is shown. Figure 8B An image displayed during a move operation is shown. Figure 8C An image displayed after the movement operation is shown. Note that the image displayed at each time is not limited to the image shown in the figure. The controller unit 300 performs image display control for displaying an image.

[0122] exist Fig. 8AIn the state shown in , a first image 801, a first information image 804, and a second information image 805 are displayed on the operation device 312. The first image 801 is an image that can be dragged and is a result of rendering by the browser processing module 250 of the image generation system 100. The first image 801 includes a first object 811 having a circular shape, a second object 812 having a triangular shape, and a third object 813 having a quadrilateral shape. In the first image 801, the positional relationship between the first object 811 to the third object 813 is fixed. The background 814 of the first image 801 is mainly white. The first information image 804 and the second information image 805 are both images whose display positions are maintained even when the first image 801 is dragged. The first information image 804 includes a forward button 821 that is operated to advance the displayed image to the next image, a back button 822 that is operated to return the displayed image to the previously displayed image, and an address bar 823 indicating the URL of the link destination of the image. A comment "Status Check" is displayed in the second information image 805.

[0123] exist Figure 8B In the state shown in , the second image 802, the first information image 804, and the second information image 805 are displayed on the operating device 312. The second image 802 includes a moving image 831 and a blank image 832. The second image 802 is generated by the browser control module 360 ​​of the image forming apparatus 101. The moving image 831 is an image indicating a state in which the first image 801 is moved by a movement amount M due to a move operation applied to the first image 801 by dragging upward. The moving image 831 includes a first object 811 whose size is reduced by the movement amount M, a second object 812 whose size is reduced by the movement amount M, and a third object 813 whose size is not changed. The background 814 is white as it is. The blank image 832 is an image that becomes blank by the movement amount M according to the movement of the first image 801. When generating the second image 802, the browser control module 360 ​​fills the blank image 832 with a color (e.g., gray) different from the color (white) of the background 814. When the movement operation of the first image 801 is performed, the second image 802 as described above is displayed. Then, the user can quickly grasp the movement amount M of the movement operation by checking the blank image 832 included in the second image 802. In the image forming apparatus 101, when the movement operation of the first image 801 is completed, the operation information about the movement amount M is sent to the image generating system 100. On the other hand, the image generating system 100 receives the operation information from the image forming apparatus 101. This enables the image generating system 100 to generate the third image 803.

[0124] exist Figure 8CIn the state shown in , the third image 803, the first information image 804 and the second information image 805 are displayed on the operating device 312. The third image 803 is an image as a result of rendering by the browser processing module 250 of the image generating system 100, that is, a changed image in a state where the first image 801 is changed according to the image moving operation. In this way, the browser processing module 250 serves as a generating unit that generates image data (generating step) of the third image 803. The image data of the third image 803 is sent to the image forming device 101. Therefore, in the image forming device 101, after the second image 802 is displayed, the second image 802 can be replaced by the third image 803. The third image 803 includes a corresponding image 841 corresponding to (equivalent to) the moving image 831 and an auxiliary image 842 that is continuously connected to the corresponding image 841 (moving image 831) and fills the blank image 832. As with the moving image 831, the corresponding image 841 includes a first object 811 whose size is reduced by a movement amount M, a second object 812 whose size is reduced by a movement amount M, and a third object 813. The auxiliary image 842 compensates the size of the third object 813 by the movement amount M. The background 814 of the third image 803 is white as it is. Note that the broken line at the boundary between the corresponding image 841 and the auxiliary image 842 in the third image 803 is for easy understanding and is not actually displayed. Then, the first image 801, the second image 802, and the third image 803 are displayed in sequence as described above, and thus the user can feel that the images have moved smoothly, that is, transitioned from the first image 801 to the third image 803.

[0125] FIG. 8D to FIG. 8G is a view showing an example of an image displayed on an operating device of the image forming apparatus. Fig.8D An image displayed before the move operation is shown. Fig. 8E and Figure 8F An image displayed during a move operation is shown. Figure 8G The image displayed after the moving operation is shown. FIG. 8A to FIG. 8C , and the description of the same matters will be omitted. Fig.8D In the state shown in , a first image 801, a first information image 804, and a second information image 805 are displayed on the operating device 312. Fig. 8E and Figure 8FIn the state shown in , a second image 802, a first information image 804, and a second information image 805 are displayed on the operating device 312. The second image 802 includes a moving image 831 and a blank image 832. In the second image 802, the display area of ​​the moving image 831 and the display area of ​​the blank image 832 are changed in conjunction with the image moving operation, respectively. Specifically, in conjunction with the image moving operation, the display area of ​​the moving image 831 gradually narrows, and correspondingly, the display area of ​​the blank image 832 gradually widens. By checking the blank image 832, the user can grasp the extent of the movement amount M at the current time when the image moving operation is performed. Figure 8G In the state shown in , a third image 803 , a first information image 804 , and a second information image 805 are displayed on the operation device 312 .

[0126] 9A to 9C is a view showing an example of an image displayed on an operating device of the image forming apparatus. Fig. 9A An image displayed before the move operation is shown. Fig. 9B An image displayed during a move operation is shown. Fig. 9C The image displayed after the moving operation is shown. FIG. 8A to FIG. 8C , and the description of the same matters will be omitted. Fig. 9A In the state shown in , the first image 801 is displayed on the operating device 312. Fig. 9B In the state shown in , the second image 802 is displayed on the operating device 312. Fig. 9C In the state shown in , the third image 803 is displayed on the operating device 312. In this way, the third image 803 is displayed on the operating device 312 FIG. 8A to FIG. 8C The image shown in does not include the first information image 804 and the second information image 805. Therefore, the entire image displayed on the operation device 312 can be dragged.

[0127] Fig.9D and Fig.9E A view used to describe the start and end points of an image that is moved by dragging. Fig.9D is a view showing the start point and the end point of the image moved by dragging upward. Fig.9E is a view showing the start point and end point of the image moved by dragging downward. Fig.9D As shown on the left side of FIG. 1 , it is assumed that the user drags upward on the operating device 312 (touch panel) by a movement amount M. In this case, the rectangular area 900 ( Fig.9D The shaded part in the figure) is transferred to the position where the starting point overlaps with the transfer destination (xdst, ydst), such as Fig.9DAs shown on the right side of . By this transfer, the rectangular area 900 is visually recognized as if it has moved upward by the movement amount M. Note that, Fig.9D The rectangular area 900 on the left side of FIG. 8 corresponds to the image data received by the receiving memory. Fig.9D The rectangular area 900 on the right side of φ corresponds to the image data received by the receiving memory and transmitted to the image memory. The width Wrect of the rectangular area 900 is obtained by (xe-xs) and is the same as the width Wdisp of the touch panel.

[0128] like Fig.9E As shown on the left side of FIG. 1 , it is assumed that the user drags downward on the operating device 312 by the movement amount M. In this case, the rectangular area 900 ( Fig.9E The shaded part in the figure) is transferred to the position where the starting point overlaps with the transfer destination (xdst, ydst), such as Fig.9E As shown on the right side of . By this transfer, the rectangular area 900 is visually recognized as if it has moved downward by the movement amount M. Note that, Fig.9E The rectangular area 900 on the left side of FIG. 8 corresponds to the image data received by the receiving memory. Fig.9E The rectangular area 900 on the right side of φ corresponds to the image data received by the receiving memory and transmitted to the image memory.

[0129] Fig. 10A and Fig. 10B is a flowchart showing processing executed by the image forming apparatus (browser control module). Fig. 10A is a flowchart showing the dragging and rendering process performed by the browser display module 363 of the browser control module 360. Fig. 10B 3 is a flowchart showing the image data receiving process performed by the image data obtaining module 361 of the browser control module 360. These processes are performed independently. Fig. 10A As shown in, in step S1001, the browser control module 360 ​​(browser display module 363) of the image forming device 101 determines the type of event such as reception of image data or user's operation (tap to enter, drag, tap to leave). This determination is based on the content of the message in the format defined between the modules. As a result of the determination in step S1001, when the event is determined to be reception of image data, the processing proceeds to step S1002. As a result of the determination in step S1001, when the event is determined to be tap to enter, the processing proceeds to step S1003. As a result of the determination in step S1001, when the event is determined to be dragging, the processing proceeds to step S1004. As a result of the determination in step S1001, when the event is determined to be tap to leave, the processing proceeds to step S1007.

[0130] In step S1002, the browser control module 360 ​​transfers the image data received in the receiving memory to the image memory, and causes the UI control module 352 to render the data in the image memory. The receiving memory is allocated to the RAM 302 (see Figure 3A ), and input image data in step S1011. Fig. 10A The broken line in indicates that image data is being read from the reception memory. Like the reception memory, the image memory is also allocated to the RAM 302. After executing step S1002, the process returns to step S1001, and the subsequent steps are executed in sequence.

[0131] In step S1003, the browser control module 360 ​​stores the coordinates (start coordinates) of the tap entry on the operation device 312 in the RAM 302. After executing step S1003, the process returns to step S1001, and the subsequent steps are sequentially executed.

[0132] In step S1004, the browser control module 360 ​​obtains data of the transmission target image (rectangular area 900) based on the start coordinates, the movement amount M, and the start point and the end point of the image data in the reception memory stored in step S1003. The method of obtaining the data is not particularly limited, and for example, a reference image may be used. Fig.9D and Fig.9E The method described. Note that the movement amount M is the difference between the coordinates stored in step S1003 and the coordinates of the drag event. After executing step S1004, the process proceeds to step S1005.

[0133] In step S1005, the browser control module 360 ​​transfers the data of the transmission target image (rectangular area 900) on the reception memory obtained in step S1004 to the image memory, and causes the UI control module 352 to render the data in the image memory. After executing step S1005, the process proceeds to step S1006.

[0134] In step S1006, the browser control module 360 ​​transfers data of a background of a different color from the moving image (image data) to an area to be a blank image in the image memory, and causes the UI control module 352 to render the blank image. The different color of the background is not particularly limited, and for example, the different color of the background may be a color obtained by inverting the original color of the area to be a blank image, or may be a predetermined shade. After executing step S1006, the process returns to step S1001, and the subsequent steps are sequentially executed.

[0135] In step S1007, the browser control module 360 ​​calculates the movement amount M based on the coordinates of the tap-in and the coordinates of the tap-out. In this embodiment, the movement amount M includes the up-down component as the moving direction of the image, but this is not restrictive. For example, when the object moves in the left-right direction, the movement amount M may include the component in the left-right direction, or may include only the component having a larger value among the component in the up-down direction and the component in the left-right direction. After executing step S1007, the process proceeds to step S1008.

[0136] In step S1008, the browser control module 360 ​​notifies the browser processing module 250 included in the image generation system 100 of the drag information. The drag information includes the start coordinates stored in step S1003 and the movement amount M obtained in step S1007. After executing step S1008, the process returns to step S1001, and the subsequent steps are sequentially executed.

[0137] like Fig. 10B As shown in FIG. 1 , in step S1010, the browser control module 360 ​​(image data acquisition module 361) of the image forming device 101 receives the image data. Then, in step S1011, the browser control module 360 ​​inputs the image data received in step S1010 into the receiving memory. As a result, the image data is stored in the receiving memory. Fig. 10B The broken lines in indicate that image data is input to the receiving memory.

[0138] In the following, reference will be made to FIG. 11A to FIG. 15 A second embodiment is described, but the differences from the above embodiment will be mainly described and the description of the same matters will be omitted. This embodiment is the same as the first embodiment except that an image that can be dragged (hereinafter, may be referred to as a "draggable image") contains a small image that can be dragged (hereinafter, may be referred to as a "frame image"). Fig.11A and Fig. 11B 1 is a sequence diagram showing an example of processing performed between the image forming apparatus and the image generating system of the cloud browser system according to the second embodiment, and an external site. Fig.11A As shown in FIG. 1 , in step S1101 , the browser processing module 250 of the image generation system 100 requests to obtain Web content from the external site 105 .

[0139] In step S1102 , the browser processing module 250 receives web content from the external site 105 in response to the request to obtain web content in step S1101 .

[0140] In step S1103a, the browser processing module 250 renders the web content received in step S1102.

[0141] In step S1103b, the browser processing module 250 detects a frame image (calculates a frame area) included in the draggable image as a result of the rendering in step S1103a.

[0142] In step S1104 a , the browser processing module 250 stores the result of the rendering performed in step S1103 a in the storage device 109 .

[0143] In step S1104b, the browser processing module 250 notifies the browser control module 360 ​​included in the image forming apparatus 101 of the URL information indicating the location of the image data of the rendering result stored in step S1104a. The browser processing module 250 also transmits the frame information about the frame image detected in step S1103b. The frame information is not particularly limited and includes, for example, the position information of the frame image and the size information of the frame image (see Fig.13A and Fig. 13B ).

[0144] In step S1105 , the browser control module 360 ​​of the image forming apparatus 101 requests acquisition of image data from the storage device 109 of the image generating system 100 based on the URL information notified in step S1104 b .

[0145] In step S1106 , the browser control module 360 ​​receives the image data from the storage device 109 as a result of the request to obtain the image data in step S1105 .

[0146] In step S1107, the browser control module 360 ​​instructs the UI control module 352 to render the image data received in step S1106. As a result, a draggable image of the image data is displayed on the operating device 312. For example, Fig. 12A The image shown in or Fig.12D Each image will be described later.

[0147] In step S1108, the user taps into the draggable image displayed on the operating device 312 in step S1107. Then, an electrical signal corresponding to the tap-in is sent from the operating device 312 to the UI control module 352. Therefore, the UI control module 352 can specify the coordinates of the position of the fingertip based on the electrical signal. Note that the tap-in of the draggable image is divided into a case where the user taps into a portion outside the frame image in the draggable image and a case where the user taps into a portion inside the frame image in the draggable image. In either case, dragging is performed after the tap-in. Dragging in the former case is different from dragging in the case described later. Fig. 11BOn the other hand, the dragging in the latter case corresponds to the steps S1111a to S1114a described later. Fig. 11B Corresponding to steps S1111b to S1114b in.

[0148] In step S1109 , the UI control module 352 of the image forming apparatus 101 transmits information on the coordinates specified in step S1108 to the browser control module 360 ​​as a tap enter event.

[0149] In step S1110 , the browser control module 360 ​​stores the information on the coordinates of the tap position (start coordinates) transmitted from the UI control module 352 in step S1109 in the RAM 302 .

[0150] In step S1111a, the user drags the fingertip that was tapped in step S1108 into the portion outside the frame image in the draggable image on the touch panel as it is. Then, an electrical signal that changes according to the dragging is sent from the operation device 312 to the UI control module 352. Therefore, the UI control module 352 can specify the coordinates that change according to the movement of the fingertip based on the electrical signal.

[0151] In step S1112a, the UI control module 352 transmits information on the coordinates specified in step S1111a and changing with the movement of the fingertip to the browser control module 360 ​​as a drag event.

[0152] In step S1113a, the browser control module 360 ​​calculates the coordinates of the start point and the end point of the drag operation in the area outside the frame image in the draggable image based on the information sent from the UI control module 352 in step S1112a. The calculation results of the browser control module 360 ​​are stored in the RAM 302.

[0153] In step S1114a, the browser control module 360 ​​instructs the UI control module 352 to render the image data based on the coordinates of the start point and the coordinates of the end point calculated in step S1113a. Thus, a partial image of the image data is displayed on the operating device 312. For example, as an image including a partial image, Fig. 12B This image will be described later.

[0154] The above steps S1111a to S1114a relate to dragging after a portion outside the frame image in a draggable image is tapped.

[0155] In step S1111b, the user drags the fingertip that tapped the portion within the frame image in the draggable image in step S1108 on the touch panel as it is. Then, an electrical signal that changes according to the dragging is sent from the operation device 312 to the UI control module 352. Therefore, the UI control module 352 can specify the coordinates that change according to the movement of the fingertip based on the electrical signal.

[0156] In step S1112b, the UI control module 352 transmits information on the coordinates specified in step S1111b and changing with the movement of the fingertip to the browser control module 360 ​​as a drag event.

[0157] In step S1113b, the browser control module 360 ​​calculates the coordinates of the start point and the end point of the drag operation in the frame image based on the information sent in step S1112b from the UI control module 352. The calculation result of the browser control module 360 ​​is stored in the RAM 302.

[0158] In step S1114b, the browser control module 360 ​​instructs the UI control module 352 to render the image data based on the coordinates of the start point and the coordinates of the end point calculated in step S1113b. Thus, a partial image of the image data is displayed on the operating device 312. For example, here is displayed Fig.12E A partial image is shown in . This image will be described later.

[0159] The above steps S1111b to S1114b are related to dragging after the frame image in the draggable image is tapped.

[0160] In step S1115, the user taps away by releasing the fingertip from the image dragged in step S1111a or S1111b. Then, an electrical signal corresponding to the tap away position is sent from the operation device 312 to the UI control module 352. Therefore, the UI control module 352 can specify the coordinates of the position where the fingertip is separated based on the electrical signal.

[0161] In step S1116 , the UI control module 352 of the image forming apparatus 101 transmits information on the coordinates specified in step S1115 to the browser control module 360 ​​as a tap-away event.

[0162] In step S1117, the browser control module 360 ​​determines whether the operation performed by the user in steps S1108 to S1115 is an operation including dragging. As described above, since dragging is performed in step S1111a or S1111b, it is determined in step S1117 that the operation includes dragging. Then, the browser control module 360 ​​calculates the movement amount of the fingertip in dragging (the scrolling amount of the image).

[0163] In step S1118 , the browser control module 360 ​​transmits information on the movement amount (scroll amount) calculated in step S1117 to the browser processing module 250 of the image generating system 100 .

[0164] In step S1119a, the browser processing module 250 performs scroll processing of generating image data based on the movement amount transmitted in step S1118.

[0165] In step S1119b, the browser processing module 250 detects a frame image included in the draggable image. Fig.14 Describe the frame image detection process.

[0166] In step S1120 a , the browser processing module 250 stores the image data generated in step S1119 a in the storage device 109 .

[0167] In step S1120b, the browser processing module 250 notifies the browser control module 360 ​​included in the image forming apparatus 101 of URL information indicating the location of the image data stored in step S1120a. The browser processing module 250 also delivers frame information about the frame image detected in step S1119b.

[0168] In step S1121 , the browser control module 360 ​​of the image forming apparatus 101 requests acquisition of image data from the storage device 109 of the image generating system 100 based on the URL information notified in step S1120 b .

[0169] In step S1122, as a result of the request to obtain image data in step S1121, the browser control module 360 ​​receives image data from the storage device 109. This image data is the image data stored in step S1120a.

[0170] In step S1123, the browser control module 360 ​​instructs the UI control module 352 to render the image data received in step S1122. As a result, an image of the image data is displayed on the operating device 312. For example, here is displayed Fig. 12C The image shown in or Fig.12F Each image will be described later.

[0171] Figures 12A to 12C is a view showing an example of an image displayed on an operating device of the image forming apparatus. Fig. 12A An image displayed before the move operation is shown. Fig. 12B An image displayed during a move operation is shown. Fig. 12C 1 shows the image displayed after the move operation. Fig. 12A In the state shown in , a first image (first draggable image) 1201 that can be dragged is displayed on the operating device 312. The first image 1201 includes frame images 1281, 1282, and 1283. Each of the frame images 1281 to 1283 includes a plurality of characters representing, for example, a program language and a message. The drag operation for each of the frame images 1281 to 1283 (the image moving operation for each frame image) may be separated from the drag operation for the first image 1201. The frame images 1281 to 1283 can be dragged independently. Therefore, the user can drag any one of the frame images 1281 to 1283, that is, the desired frame image. The number of frame images arranged in the first image 1201 in this embodiment is three, but is not limited thereto, and may be at least one. The first image 1201 includes a first object 1211 having a circular shape, a second object 1212 having a triangular shape, and a third object 1213 having a quadrilateral shape. The background 1214 of the first image 1201 is mainly white. In the first image 1201, the positions of frame images 1281, 1282, and 1283 are respectively fixed to the positions of the first object 1211, the second object 1212, and the third object 1213. The first image 1201 is an image obtained as a result of rendering by the browser processing module 250 of the image generation system 100.

[0172] exist Fig. 12BIn the state shown in , the second image 1202 is displayed on the operating device 312. The second image 1202 includes a moving image 1231 and a blank image 1232. The second image 1202 is generated by the browser control module 360 ​​of the image forming apparatus 101. The moving image 1231 is an image indicating a state in which the first image 1201 is moved by a movement amount M when a move operation is applied to a portion (e.g., background 1214) other than the frame images 1281 to 1283 in the first image 1201 by dragging. The moving image 1231 includes the frame image 1281 whose size is reduced by the movement amount M, the frame image 1282 whose size is not changed, and the frame image 1283 whose size is reduced by the movement amount M. The moving image 1231 includes the second object 1212 whose size is reduced by the movement amount M and the third object 1213 whose size is not changed. In the moving image 1231, the first object 1211 has disappeared. The background 1214 is white as it is. The blank image 1232 is an image that becomes blank according to the movement of the first image 1201 by the movement amount M. When generating the second image 1202, the browser control module 360 ​​fills the blank image 1232 with a color different from the color of the background 1214. When the movement operation of the first image 1201 is performed, the second image 1202 as described above is displayed. Then, the user can quickly grasp the movement amount M of the movement operation by checking the blank image 1232 included in the second image 1202.

[0173] exist Fig. 12CIn the state shown in , a third image (third draggable image) 1203 is displayed on the operating device 312. The third image 1203 is an image as a result of rendering by the browser processing module 250 of the image generating system 100. The image data of the third image 1203 is sent from the browser processing module 250 to the image forming apparatus 101. Therefore, in the image forming apparatus 101, after the second image 1202 is displayed, the second image 1202 can be replaced by the third image 1203. The third image 1203 includes a corresponding image 1241 corresponding to the moving image 1231 and an auxiliary image 1242 that is continuously connected to the corresponding image 1241 and fills the blank image 1232. Like the moving image 1231, the corresponding image 1241 includes a frame image 1281 whose size is reduced by a movement amount M, a frame image 1282 whose size is not changed, and a frame image 1283 whose size is reduced by a movement amount M. In addition, like the moving image 1231, the moving image 1241 includes the second object 1212 whose size is reduced by the movement amount M and the third object 1213 whose size is not changed. The auxiliary image 1242 compensates the size of the third object 1213 by the movement amount M. The background 1214 of the third image 1203 is white as it is. The first image 1201, the second image 1202, and the third image 1203 are sequentially displayed as described above, and thus the user can feel that the images have been smoothly moved, that is, transitioned from the first image 1201 to the third image 1203.

[0174] FIG. 12D to FIG. 12F is a view showing an example of an image displayed on an operating device of the image forming apparatus. Fig.12D An image displayed before the move operation is shown. Fig.12E An image displayed during a move operation is shown. Fig.12F 1 shows the image displayed after the move operation. Fig.12D In the state shown in Fig. 12A Similarly, a first image 1201 that can be dragged is displayed on the operating device 312. The first image 1201 includes frame images 1281, 1282, and 1283 and a first object 1211, a second object 1212, and a third object 1213. Here, as an example, the frame image 1281 among the frame images 1281 to 1283 is selected as the drag target image. Therefore, the frame image 1281 in the first image 1201 is a frame image (first frame image) before dragging.

[0175] exist Fig.12EIn the state shown in , the second image 1222 is displayed on the operating device 312. The second image 1222 is generated by the browser control module 360 ​​of the image forming apparatus 101. The second image 1222 includes a frame image 1251. The frame image 1251 is an image obtained as a result of applying a move operation to the frame image 1281. Like the first image 1201, the second image 1222 includes a frame image 1282, a frame image 1283, and a first object 1211, a second object 1212, and a third object 1213. The frame image 1251 includes a moving image (moving small image) 1252 and a blank image (blank small image) 1253. The moving image 1252 is an image indicating a state in which the frame image 1281 is moved by a movement amount M due to a move operation applied to the frame image 1281 by dragging upward. In the moving image 1252, the number of characters (the number of lines) is reduced by the movement amount M. The blank image 1253 is an image that becomes blank according to the movement of the frame image 1281 by the movement amount M. When generating the second image 1222, the browser control module 360 ​​fills the blank image 1253 with, for example, a color different from the color of the moving image 1252 and the color of the background 1214. Then, the user can quickly grasp the movement amount M of the move operation by checking the blank image 1253 included in the second image 1222.

[0176] exist Fig.12F In the state shown in , a third image (third draggable image) 1223 is displayed on the operating device 312. The third image 1223 is an image as a result of rendering by the browser processing module 250 of the image generating system 100. The image data of the third image 1223 is sent from the browser processing module 250 to the image forming apparatus 101. Therefore, in the image forming apparatus 101, after the second image 1222 is displayed, the second image 1222 can be replaced with the third image 1223. The third image 1223 includes a frame image 1261 instead of the frame image 1251 in the second image 1222. Like the second image 1222, the third image 1223 includes a frame image 1282, a frame image 1283, and the first object 1211, the second object 1212, and the third object 1213. The frame image 1261 includes a corresponding image 1262 corresponding to the moving image 1252 of the frame image 1251 and an auxiliary image 1263 that is continuously connected to the corresponding image 1262 and fills the blank image 1253. In the corresponding image 1262, the number of characters (the number of rows) is reduced by the movement amount M as in the moving image 1252. In the auxiliary image 1263, the number of characters (the number of rows) lacking the movement amount M is newly supplemented. The first image 1208, the second image 1222, and the third image 1223 are displayed in sequence, and thus the user can feel that the frame image smoothly transitions.

[0177] Fig.13A and Fig. 13B is a view showing an example of frame information related to a frame image. The frame information is configured in JSON format. Fig.13A A general example of framework information is shown. Fig. 13B A specific example of framework information is shown. Fig.13A As shown in , when the number of frame images is N (N is an integer), the upper left corner coordinates (xi, yi), width (width_i) and height (height_i) of each frame image are encoded in JSON format as frame information. Note that "i" is an integer from "1" to "N". Fig. 13B The frame information in the case where there are three frame images is shown. For example, the upper left corner coordinates (3, 35), width (130), and height (600) of the first frame image are encoded in the JSON format as the frame information.

[0178] Fig.14 is a flowchart showing the processing performed by the image generation system (browser processing module). Fig.14 The flowchart shown in FIG. 1 shows the process of detecting the frame image. Fig.14 As shown in FIG. 14 , in step S1401, the browser processing module 250 extracts, using JavaScript for example, all iframe tag parts indicating frame images in an HTML document as web contents received from the external site 105. For example, the iframe tag parts are described as follows.

[0179] <iframe src＝”http: hogehoge.com ”>< / iframe>

[0180] In step S1402, the browser processing module 250 detects "scroll" and "auto" included in the style "overflow" of CSS (Cascading Style Sheets) from the web content. The term "scroll" refers to dragging in a draggable (scrollable) area. "Auto" indicates that "scroll" or other operations can be performed. When "scroll" or "auto" is detected, dragging on the frame image is enabled. It is noted that it is not necessary to detect "scroll" and "auto" as long as dragging on the screen can be performed.

[0181] In step S1403, the browser processing module 250 detects the upper left corner coordinates (xi, yi), width (width_i) and height (height_i) of the iframe tag part (frame image) extracted in step S1401 as frame information. This frame information is encoded in JSON format (see Fig.13A and Fig. 13B ).

[0182] Fig.15 is a flowchart showing processing executed by the image forming apparatus (browser control module). Fig.15 Steps S1501 to S1503 of the flowchart shown in FIG. Fig. 10A Steps S1001 to S1003 of the flowchart shown in are the same. In addition, steps S1511 and S1512 are the same as steps S1007 and S1008.

[0183] like Fig.15 As shown in , in step S1504 after executing step S1501, the browser control module 360 ​​determines whether the coordinates (tap-enter start coordinates) stored in step S1503 are included in any frame image within the draggable image. In this way, the browser control module 360 ​​has a function as a determination unit that determines whether the drag after the tap-enter on the operating device 312 is a moving operation of the draggable image or a moving operation of the frame image. Then, subsequent control is performed based on the determination result. The determination in step S1504 is based on whether the start coordinates are included in the area defined by the upper left corner coordinates (xi, yi), width (width_i), and height (height_i) of the frame information. Then, as a result of the determination in step S1504, when it is determined that the start coordinates are included in the frame image, the processing proceeds to step S1505. On the other hand, as a result of the determination in step S1504, when it is determined that the start coordinates are not included in the frame image (that is, the start coordinates are outside the frame image), the processing proceeds to step S1508.

[0184] In step S1505, as in step S1004, the browser control module 360 ​​obtains the data of the transmission target image in the frame image based on the start coordinates, the movement amount M, and the start point and end point of the image data in the reception memory stored in step S1503. The width of the transmission target image is the same as the width of the frame image. After executing step S1505, the process proceeds to step S1506.

[0185] In step S1506, the browser control module 360 ​​transfers the data of the transmission target image in the frame image in the reception memory obtained in step S1505 to the image memory, and causes the UI control module 352 to render the data in the image memory. After executing step S1506, the process proceeds to step S1507.

[0186] In step S1507, the browser control module 360 ​​transfers data of a background different from the moving image of the frame image to an area on the image memory to be a blank image of the frame image, and causes the UI control module 352 to render the blank image of the frame image. After executing step S1507, the process returns to step S1501, and the subsequent steps are executed in sequence.

[0187] In step S1508 , the browser control module 360 ​​obtains draggable image data based on the start coordinates, the movement amount M, and the start point and the end point of the image data in the reception memory stored in step S1503 .

[0188] In step S1509, the browser control module 360 ​​transfers the draggable image data in the reception memory obtained in step S1508 to the image memory, and causes the UI control module 352 to render the data in the image memory. After executing step S1509, the process proceeds to step S1510.

[0189] In step S1510, the browser control module 360 ​​transfers data of a background different from the moving image (image data) to an area in the image memory to be a blank image, and causes the UI control module 352 to render the blank image. After executing step S1510, the process returns to step S1501, and the subsequent steps are executed in sequence.

[0190] In the following, reference will be made to FIG. 16A to FIG. 19 The third embodiment is described, but the differences from the above-described embodiments will be mainly described and description of the same matters will be omitted. This embodiment is the same as the first embodiment except that the processing performed by the image generation system is different. FIG. 16A to FIG. 16D : is a view showing an example of an image displayed on an operating device of an image forming apparatus of a cloud browser system according to the third embodiment. Fig.16A An image displayed before the move operation is shown. Fig. 16B An image displayed during a move operation is shown. Fig. 16C Shown in the display Fig. 16B The image shown in the image after the image displayed during the move operation. Fig.16D is the image displayed after the move operation. Fig. 8A , Figure 8B and Figure 8C Description of the differences and similarities. Fig.16A In the state shown in , a first image 801 , a first information image 804 , and a second information image 805 are displayed on the operation device 312 . Fig.16A The status shown in Fig. 8A The same state as shown in Fig. 16BIn the state shown in , the second image 802 , the first information image 804 , and the second information image 805 are displayed on the operation device 312 . Fig. 16C The status shown in Figure 8B The same state as shown in Fig.16D , a third image 803 , a first information image 804 , and a second information image 805 are displayed on the operation device 312 . Fig.16D The status shown in Figure 8C Same status as shown in . Fig. 16C The status shown in FIG. 8A to FIG. 8C The status shown in is different.

[0191] exist Fig. 16C In the state shown in , the intermediate image 801a, the first information image 804, and the second information image 805 are displayed on the operating device 312. The intermediate image 801a is an image displayed next to the second image 802. The first image 801a shows Fig. 16B 801a is displayed when the image forming apparatus 101 does not wait for the reception of the image data of the third image 803 generated by the image generating system 100 or when the reception is insufficient. In addition, the intermediate image 801a is displayed when the image data of the third image 803 is insufficiently transmitted from the image generating system 100.

[0192] Therefore, the image generation system 100 is configured to suppress the phenomenon of displaying the intermediate image 801a. The configuration and operation will be described below. Fig.17 is a sequence diagram showing an example of processing performed between the image forming apparatus, the image generating system, and the external site. Fig.17 Steps S1701 to S1718 in the sequence diagram shown in FIG. Figure 7 Steps S701 to S718 in the sequence diagram shown in FIG. 1 are the same. After executing step S1718, step S3001, step S1719, and step S3002 are executed in sequence. Step S1719 is the same as Figure 7 After executing step S3002, steps S1720a to S1723 are executed in sequence. Steps S1720a to S1723 are the same as step S719 in the sequence diagram shown in FIG. Figure 7 Steps S720a to S723 in the sequence diagram shown in are the same.

[0193] In step S3001 , the browser processing module 250 receives operation information from the image forming apparatus 101 , and determines whether the operation information is information related to an image moving operation (scroll operation) for moving an image displayed on the operating device 312 of the image forming apparatus 101 .

[0194] In step S3002, the browser processing module 250 performs a rendering period determination process (determination step) of determining whether the generation of the third image 803 in step S1719 is completed. Therefore, in this embodiment, the browser processing module 250 also has a function as a determination unit that determines whether the generation of the third image 803 is completed. Note that step S3002 is executed every time rendering is performed in step S1719.

[0195] Fig.18 FIG. 1 is a flowchart showing the processing performed in the image generation system. Fig.18 As shown in , in step S2001, the browser processing module 250 determines whether the operation information from the image forming apparatus 101 is related to a scroll operation of scrolling (dragging) an image displayed on the operation device 312 of the image forming apparatus 101. As a result of the determination in step S2001, when it is determined that the information is related to the scroll operation, the process proceeds to step S2002. On the other hand, as a result of the determination in step S2001, when it is determined that the information is not related to the scroll operation, the process proceeds to step S2003.

[0196] In step S2002, the browser processing module 250 turns on the scroll mode. This enables the start of generation of an image displayed after the scroll operation (ie, image data of the third image 803) (step S1719).

[0197] In step S2003, the browser processing module 250 turns off the scroll mode. This omits the generation of the image data of the third image 803 (step S1719).

[0198] In step S2004, the browser processing module 250 executes a process of generating image data of the third image 803 as a process based on the operation information (step S1719).

[0199] Fig.19 is a diagram showing the image generation system in relation to Fig.18 Flowchart of the process shown in FIG. 1 is a flowchart of the process executed at different times. Fig.19As shown in , in step S2101, the browser processing module 250 of the image generation system 100 determines whether the scroll mode is turned on. As a result of the determination in step S2101, when it is determined that the scroll state is turned on, the process proceeds to step S2102. On the other hand, as a result of the determination in step S2101, when it is determined that the scroll mode is not turned on (that is, the scroll mode is turned off), the process proceeds to step S2104.

[0200] In step S2102, the browser processing module 250 determines whether a predetermined time period has passed from the start of generation (rendering) of the image data of the third image 803. As a result of the determination in step S2102, when it is determined that the predetermined time period has passed, the processing proceeds to step S2103. On the other hand, as a result of the determination in step S2102, when it is determined that the predetermined time period has not passed, the processing stays in step S2102, that is, the processing waits until the predetermined time period has passed. The predetermined time period is appropriately set according to, for example, the processing speed of the browser processing module 250, but is not particularly limited. The predetermined time period is preferably changeable.

[0201] In step S2103, the browser processing module 250 turns off the scroll mode. Therefore, it is determined that the generation of the image data of the third image 803 is completed.

[0202] In step S2104, the browser processing module 250 stores the image data of the third image 803 as the rendering result in the storage device 109 (step S1720a).

[0203] In step S2105, the browser processing module 250 notifies the browser control module 360 ​​included in the image forming apparatus 101 of the URL information indicating the location of the image data stored in step S2104 (step S1720b). After executing step S2105, the process is terminated.

[0204] As described above, in this embodiment, the image generation system 100 waits for the transmission of the image data of the third image 803 to the image forming apparatus 101 until it is determined that the generation of the image data of the third image 803 is completed. Then, when it is determined that the generation of the image data of the third image 803 is completed, the image data of the third image 803 (in this embodiment, a notification of the URL information) is transmitted to the image forming apparatus 101. Thus, the phenomenon of displaying the intermediate image 801a is suppressed, and the first image 801 is sequentially displayed on the operating device 312 of the image forming apparatus 101 (see Fig.16A ), the second image 802 (see Fig. 16B ) and the third image 803 (see Fig.16D). Such an image change allows the user to feel that the image on the operating device 312 has moved smoothly and successfully.

[0205] In the following, reference will be made to Fig. 20 A fourth embodiment is described, but differences from the above-described embodiments will be mainly described and description of the same matters will be omitted. This embodiment is the same as the third embodiment except that the processing performed by the image generation system is different. Fig. 20 4 is a flowchart showing the processing performed by the image generation system of the cloud browser system according to the fourth embodiment. Fig. 20 In the flowchart shown in , step S2201 and step S2202 are performed in sequence, instead of Fig.19 Step S2102 in the flowchart shown in FIG. Fig. 20 As shown in , as a result of the determination in step S2101, when it is determined that the scroll state is on, the process proceeds to step S2201.

[0206] In step S2201, the browser processing module 250 of the image generation system 100 stores the rendering result in, for example, the storage device 109. The rendering result stored here is the result of the rendering performed in step S1719, and in this embodiment is the image data of the two third images 803 generated at different times. The image data of the two third images 803 can be, for example, image data obtained within a predetermined period of time.

[0207] In step S2202, the browser processing module 250 compares the image data (rendering results) of the two third images 803 stored in step S2201, and determines whether the number of differences between the image data of the two images is equal to or less than a threshold value. As a result of the determination in step S2202, when it is determined that the number of differences is equal to or less than the threshold value, the processing proceeds to step S2103. As described above, in step S2103, the scroll mode is turned off and the generation of the image data of the third image 803 is determined to be completed. On the other hand, as a result of the determination in step S2202, when it is determined that the number of differences is greater than the threshold value, the processing terminates. Although the image data of the two third images 803 are compared in step S2202 in this embodiment to find differences, this is not restrictive. For example, the image data of three or more third images 803 can be compared. The threshold value is, for example, pre-stored in the storage device 109. The threshold value is preferably appropriately changeable.

[0208] As in the third embodiment, in this embodiment, the image generation system 100 waits for the image data of the third image 803 to be sent to the image forming apparatus 101 until it is determined that the generation of the image data of the third image 803 is completed. Then, when it is determined that the generation of the image data of the third image 803 is completed, the image data of the third image 803 is sent to the image forming apparatus 101. Thus, the phenomenon of displaying the intermediate image 801a is suppressed, and the first image 801 to the third image 803 are sequentially displayed on the operation device 312 of the image forming apparatus 101. Such image changes allow the user to feel that the image on the operation device 312 has been smoothly and successfully moved.

[0209] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications and changes can be made within the scope of the gist of the present invention. In addition, in each embodiment, the information processing device is the image forming device 101, but it is not limited to this, and can be, for example, a desktop or notebook personal computer, a smart phone, etc.

[0210] In the cloud browser system 1000, for example, the image generation system 100 as a server can be installed outside Japan, and the image forming device 101 as a terminal device can be installed within Japan. Even in such a case, the server can send files and data to the terminal device, and the terminal device can also receive files and data. Even when the server is located outside Japan, the cloud browser system 1000 can send and receive files and data as a unit. In the cloud browser system 1000, even when the server is located outside Japan and the terminal device is located in Japan, the terminal device can perform the main functions of the cloud browser system 1000. In addition, the effect of the function can be presented within Japan. For example, even if the server is located outside Japan, if the terminal device constituting the cloud browser system 1000 is located in Japan, the system can be used in Japan by using the terminal device.

[0211] Other embodiments

[0212] Embodiments of the present invention may also be implemented by reading and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a 'non-transitory computer-readable storage medium') to perform one or more of the functions of the above-mentioned embodiments and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more of the functions of the above-mentioned embodiments, and by a computer of the system or device, for example, reading and executing computer executable instructions from a storage medium to perform one or more of the functions of the above-mentioned embodiments and / or controlling one or more circuits to perform one or more of the functions of the above-mentioned embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)) TM ), one or more of flash memory devices, memory cards, etc.

[0213] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0214] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0215] This application claims the benefit of Japanese Patent Application No. 2023-204679 filed on December 4, 2023 and Japanese Patent Application No. 2024-064559 filed on April 12, 2024, which are hereby incorporated by reference herein in their entirety.

Claims

1. An information processing system, comprising: An information processing device comprising: a display device that displays image data obtained from a server communicably connected to the information processing apparatus as an image; an operating device that accepts a user's movement operation for moving an image displayed on the display device; a device-side sending unit configured to send operation information about a movement amount of the moving operation to the server after the moving operation ends; and a control unit configured to control the display device to display the image before the moving operation as a first image, and to display a second image when the moving operation is applied to the first image, the second image including a moving image in a state where the first image is moved by the moving operation and a blank image that becomes blank according to the movement of the first image; and The server comprises: a server-side receiving unit, wherein the server-side receiving unit is configured to receive the operation information from the device-side sending unit; a generating unit configured to generate image data of a third image based on the operation information received by the server-side receiving unit, the third image including an image corresponding to the moving image and an image continuously connected to the moving image and filling the blank image; and a server-side sending unit configured to send the image data of the third image to the information processing device, and The control unit controls the display device to replace the second image with the third image after displaying the second image.

2. The information processing system according to claim 1, wherein the first image includes at least one small image, the small image moving operation of the operating device on the at least one small image is separably applicable from the moving operation on the first image, and The control unit controls the display device to display the small image before the small image moving operation as the first small image in the first image, and to display the second small image in the first image when the small image moving operation is applied to the first small image, wherein the second small image includes a moving small image in a state where the first small image is moved by the small image moving operation and a blank small image that becomes blank according to the movement of the first small image.

3. The information processing system according to claim 2, wherein the generating unit generates image data of a third small image, the third small image including an image corresponding to the moving small image and an image continuously connected to the moving small image and filling the blank small image, and The control unit controls the display device to replace the second small image with the third small image after displaying the second small image. 4 . The information processing system according to claim 2 , wherein the first image includes a plurality of small images, and the small image moving operation is independently applicable to the plurality of small images, respectively.

5. The information processing system according to claim 2 or 3, wherein the information processing device comprises a determination unit configured to determine whether the move operation is applied to the first image and whether the small image move operation is applied to the small image, and The control unit performs control based on a determination result of the determination unit.

6. The information processing system according to any one of claims 1 to 3, wherein the moving image includes a background, The control unit fills the blank image with a color different from a main color of the background. 7 . The information processing system according to claim 1 , wherein the control unit changes a display area of ​​the moving image and a display area of ​​the blank image in conjunction with the movement operation.

8. The information processing system according to claim 7, wherein the control unit narrows a display area of ​​the moving image in conjunction with the movement operation, and widens a display area of ​​the blank image corresponding to the narrowing of the display area of ​​the moving image.

9. The information processing system according to any one of claims 1 to 3 and 8, wherein the display device includes a touch panel having a touch function, and The operating device is implemented by the touch function. 10 . The information processing system according to claim 9 , wherein the movement operation is performed by dragging by sliding a finger on the touch panel.

11. The information processing system according to any one of claims 1 to 3, 8 and 10, wherein the information processing apparatus includes a storage device that stores image data from the server.

12. The information processing system according to any one of claims 1 to 3, 8 and 10, wherein the display device displays an image of a website as the image.

13. The information processing system according to any one of claims 1 to 3, 8 and 10, wherein the information processing apparatus is an image forming apparatus having at least one of a printing function, a scanning function, and a facsimile function.

14. A server communicatively connected to an information processing device, the server comprising: a server-side receiving unit configured to receive operation information about a moving operation for moving an image displayed on a display device of the information processing apparatus, the operation information being transmitted from the information processing apparatus; a generating unit configured to generate image data of a changed image in a state changed according to the moving operation based on the operation information received by the server-side receiving unit; a server-side sending unit, the server-side sending unit being configured to send the image data of the changed image to the information processing device; as well as a determining unit configured to determine whether the generation of the image data of the changed image by the generating unit is completed; as well as The server-side transmission unit waits for transmission of the image data of the changed image to the information processing device until the determination unit determines that generation of the image data of the changed image is completed. 15 . The server according to claim 14 , wherein the determination unit determines that the generation of the image data of the change image by the generation unit is completed when a predetermined period of time has passed from the start of the generation of the image data of the change image by the generation unit.

16. The server according to claim 14, wherein the determination unit determines that the generation of the image data of the change image by the generation unit is completed when the number of differences between the image data of the plurality of change images generated by the generation unit at different times is equal to or smaller than a threshold value.

17. A control method for an information processing system, the information processing system comprising an information processing device and a server communicatively connected to the information processing device, the control method comprising: a first image display step of displaying the image data obtained from the server as a first image on a display device provided in the information processing apparatus; an operation receiving step, in which an operation device provided in the information processing apparatus receives a user's movement operation for moving the first image displayed on the display device; a device-side sending step of sending operation information about a moving amount of the moving operation from the information processing device to the server after the moving operation is completed; a second image display step of displaying a second image on the display device, the second image including a moving image in a state where the first image is moved by the moving operation and a blank image that becomes blank according to the movement of the first image; A server-side receiving step, wherein the server receives the operation information from the device-side sending unit; a generating step, wherein the server generates image data of a third image based on the operation information received by the server-side receiving unit, the third image including an image corresponding to the moving image and an image continuously connected to the moving image and filling the blank image; A server-side sending step of sending the image data of the third image to the information processing device; as well as A third image display step of displaying the third image based on the image data received from the server on the display device in place of the second image.

18. A control method for a server communicably connected to an information processing device, the control method comprising: A server-side receiving step of receiving operation information about a moving operation for moving an image displayed on a display device of the information processing apparatus, the operation information being transmitted from the information processing apparatus; a generating step of generating image data of a changed image in a state changed according to the moving operation based on the operation information received in the receiving step on the server side; a determining step of determining whether the generation of the image data of the changed image in the generating step is completed; as well as A server-side sending step of sending the image data of the changed image to the information processing device after it is determined in the determining step that the generation of the image data of the changed image is completed.

19. A computer program product comprising a control program for causing a computer to execute the unit of the information processing system according to any one of claims 1 to 13.

20. A computer program product comprising a control program causing a computer to execute the unit of the server according to any one of claims 14 to 16. 21 . A non-transitory computer-readable storage medium storing a control program that causes a computer to execute the unit of the information processing system according to claim 1 .

22. A non-transitory computer-readable storage medium storing a control program for causing a computer to execute the unit of the server according to any one of claims 14 to 16.

Images