Image Generation Device

Abstract

Allows the transmission of all image information based on the rendering result of a web page. [Solution] The image generating device has a rendering means that renders a web page when it receives operation information for the web page, an acquisition means that acquires a difference area between image information resulting from the current rendering by the rendering means and image information resulting from the previous rendering, a first generating means that generates image information of a single rectangular area including the difference area when it receives an image information request, and a transmitting means that transmits the image information of the single rectangular area generated by the first generating means.

Description

[Technical Field]

[0001] The present disclosure relates to an image generating device, an information processing system, a processing method for an image generating device, and a program. [Background technology]

[0002] Communication devices equipped with web browsers (hereafter referred to as browsers) that can view web pages are becoming widespread. By displaying web pages of external services through the browser, communication devices can link with external services.

[0003] One type of browser is called a cloud browser, in which a browser engine that generates screen data for a web page is placed on a cloud server. A cloud browser is a method for viewing a web page by sending screen data drawn by the browser engine on the cloud server to a communication device and displaying it on the communication device. Because the browser engine handles high-load processes such as analyzing and executing web pages, using a cloud browser can reduce the load on the communication device.

[0004] In a cloud browser, screen data is frequently communicated between the cloud and a communication device, so it is necessary to reduce the amount of communication bandwidth consumed by reducing the amount of screen data transmitted.

[0005] As a method for reducing the amount of screen data, for example, Patent Document 1 discloses a technology in which the latest screen data is generated at predetermined intervals and differential information between the previous screen data and the latest screen data is transmitted to a communication device. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-33790 A Summary of the Invention [Problem to be solved by the invention]

[0007] Depending on the load state, a communication device may be unable to display screen data even if it receives it. In this case, as in Patent Document 1, when the latest screen data is generated at a predetermined interval and difference information is transmitted, if the communication device is unable to receive or display the screen data, the communication device is unable to acquire the difference information. In this case, even if the next transmitted difference information can be displayed, there is a problem that the screen data on the cloud server may differ from the previous difference information because the previous difference information cannot be displayed.

[0008] The objective of this disclosure is to enable the transmission of all image information based on the result of rendering a web page. [Means for solving the problem]

[0009] The image generating device includes a rendering means for rendering a web page when it receives operation information for the web page, an acquisition means for acquiring a difference area between image information resulting from the current rendering by the rendering means and image information resulting from the previous rendering, a first generation means for generating image information of a single rectangular area including the difference area when it receives an image information request, and a transmission means for transmitting the image information of the single rectangular area generated by the first generation means. [Effects of the Invention]

[0010] According to the present disclosure, all image information based on the rendering result of a web page can be transmitted. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an example of a system configuration. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of a communication device. [Figure 3] FIG. 2 illustrates an example of a functional configuration of a communication device. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image generating apparatus. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional configuration of an image generating apparatus. [Figure 6] FIG. 10 is a diagram illustrating an example of a rendering result. [Figure 7] FIG. 10 is a diagram illustrating an example of difference information. [Figure 8] FIG. 10 is a diagram showing an example of image information generated by the maximum rectangle method. [Figure 9] 10 is a flowchart illustrating a processing flow of the communication device. [Figure 10] 10 is a flowchart illustrating a processing flow of an image generation device and a browser engine. [Figure 11] FIG. 10 is a diagram illustrating an example of a rendering result. [Figure 12] FIG. 10 is a diagram illustrating an example of difference information. [Figure 13] FIG. 10 is a diagram showing an example of image information generated by the maximum rectangle method. [Figure 14] FIG. 10 is a diagram showing an example of image information generated by a differential domain method. [Figure 15] 10A and 10B are diagrams illustrating an example of screen division using a cell comparison method and generated image information. [Figure 16] 10 is a flowchart illustrating a processing flow of an image generation device and a browser engine. [Figure 17] 10 is a flowchart illustrating a processing flow of an image generation device and a browser engine. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solution. Note that the same components will be described with the same reference numerals.

[0013] (First embodiment) 1 is a diagram showing an example of the configuration of a system 110 according to the first embodiment. The system 110 includes an information processing system 100 and a Web server 102. The information processing system 100 includes a communication device 101 and an image generation device 103.

[0014] A user of the communication device 101 views and operates the screen of the communication device 101. The communication device 101 is connected to the image generation device 103 via a network. The communication device 101 transmits operation information for a web page to the image generation device 103, thereby operating the web page on the image generation device 103.

[0015] The network in this embodiment may be, for example, a combination of the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), etc. Furthermore, a plurality of communication devices 101 may exist.

[0016] The web server 102 is connected to the image generating device 103 via a network. Web content is stored on the web server 102. In this embodiment, the web content refers to HTML, CSS, JavaScript (registered trademark), image files such as jpeg and png, and video files such as mp4 and webm.

[0017] The image generation device 103 is equipped with a browser engine 104. The image generation device 103 is connected to the communication device 101 and the Web server 102 via a network. The image generation device 103 receives Web operation information from the communication device 101. The image generation device 103 also acquires Web content from the Web server 102.

[0018] The browser engine 104 runs on the image generation device 103 and analyzes and renders the Web content.

[0019] 2 is a diagram showing an example of the hardware configuration of a computer device that constitutes the communication device 101 of this embodiment. The communication device 101 includes a CPU 201, a ROM 202, a RAM 203, a graphical interface 204, and a network interface 205.

[0020] The CPU 201 is a central processing unit (CPU) that controls the communication device 101 .

[0021] The ROM 202 is a read-only memory (ROM) that stores programs and parameters that do not require modification.

[0022] The RAM 203 is a random access memory (RAM) that temporarily stores programs and data supplied from external devices.

[0023] The graphical interface 204 is a graphical interface that includes a screen for displaying stored data and supplied data, a pointing device such as a mouse for inputting data in response to user operations, an input device such as a keyboard, and a touch panel.

[0024] The network interface 205 is a network interface for connecting to a network.

[0025] Also, there may be a hard disk or memory card fixedly installed in the communication device 101. Alternatively, there may be an external storage device including an optical disk such as a flexible disk (FD) or compact disk (CD) that is detachable from the communication device 101, a magnetic or optical card, an IC card, a memory card, or the like.

[0026] Fig. 3 is a diagram showing an example of the functional configuration of a computer device that constitutes the communication device 101 of this embodiment. The communication device 101 has a communication unit 301, a storage unit 302, an operation detection unit 303, a screen generation unit 304, a response analysis unit 305, and an application unit 306. The CPU 201 in Fig. 2 executes a program to realize each of the mechanism components in Fig. 3.

[0027] The communication unit 301 controls the communication performed by the communication device 101 via the network interface 205 .

[0028] The storage unit 302 stores information, such as image information, transmission switching conditions, switching conditions to the polling method, the transmission method, and a rendering flag.

[0029] The operation detection unit 303 detects the content of the operation when the graphical interface 204 is operated.

[0030] The screen generation unit 304 generates and displays the screen to be displayed on the graphical interface 204 .

[0031] The response analysis unit 305 analyzes the contents of the operation information result received by the communication unit 301 from the image generation device 103 .

[0032] The application unit 306 starts and ends the application for performing processing as a cloud browser.

[0033] 4 is a diagram showing an example of the hardware configuration of a computer device that constitutes the image generation device 103. The image generation device 103 has a CPU 401, a ROM 402, a RAM 403, and a network interface 404.

[0034] The CPU 401 is a central processing unit (CPU) that controls the image generating device 103 .

[0035] The ROM 402 is a read-only memory (ROM) that stores programs and parameters that do not require modification.

[0036] The RAM 403 is a random access memory (RAM) that temporarily stores programs and data supplied from external devices.

[0037] The network interface 404 is a network interface for connecting to a network.

[0038] 5 is a diagram showing an example of the functional configuration of the image generation device 103 of this embodiment. The image generation device 103 has a communication unit 501, a storage unit 502, a difference information storage unit 503, a rendering determination unit 504, a request determination unit 505, a browser application unit 506, a difference information calculation unit 507, an image information generation unit 508, and an image information generation means determination unit 509. The image generation device 103 also has a browser engine 104. The browser engine 104 has a Web content analysis unit 510 and a Web content drawing unit 511. The CPU 401 in FIG. 4 executes a program to realize each of the functional configuration units in FIG. 5.

[0039] The communication unit 501 controls communication performed by the image generation device 103 via the network interface 404. The communication unit 501 also performs communication between the image generation device 103 and the browser engine 104.

[0040] The storage unit 502 stores image information of the rendering results and the upper limit of the number of images.

[0041] The difference information storage unit 503 stores the difference information.

[0042] The rendering determination unit 504 determines whether rendering has been performed.

[0043] The request determination unit 505 determines the content that the communication unit 501 has received from the communication device 101. The request determination unit 505 also determines whether the communication unit 501 has received operation information from the communication device 101.

[0044] The browser application unit 506 starts and ends the browser application for performing processing as a cloud browser.

[0045] The difference information calculation unit 507 analyzes the difference information stored in the difference information storage unit 503 .

[0046] The image information generating unit 508 generates image information from the difference information stored in the difference information storage unit 503 .

[0047] The image information generating means determining unit 509 determines the image information generating means of the image information generating unit 508 .

[0048] The Web content analysis unit 510 analyzes Web content. The Web content analysis unit 510 also determines whether operation information has been received from the image generation device 103, determines when the Web content's drawing reflection process over time has occurred, performs Web operations on the analyzed Web content, and transmits a drawing completion notification. The Web content analysis unit 510 also performs operation information reflection processing and drawing reflection processing over time.

[0049] The Web content drawing unit 511 draws the analyzed Web content.

[0050] FIG. 6 is a diagram showing an example of the result of rendering by the Web content rendering unit 511. As shown in FIG.

[0051] The result 601 is an example of the result of initial rendering by the Web content rendering unit 511 when the communication device 101 starts an application in the application unit 306 .

[0052] Result 602 is an example of the result of the next rendering performed by communication unit 501 after Web content rendering unit 511 has rendered result 601 and communication unit 501 has received operation information from communication apparatus 101 .

[0053] Result 603 is an example of the result of the next rendering performed after the Web content rendering unit 511 has rendered result 602.

[0054] Result 604 is an example of the result of the next rendering performed after the Web content rendering unit 511 has rendered result 603.

[0055] 7 is a diagram showing an example of difference information stored in the difference information storage unit 503 of this embodiment. Difference information 701 to 703 indicate the state of difference information stored in the difference information storage unit 503.

[0056] The difference information 701 to 703 includes a difference area, which indicates an area where the Web content rendering unit 511 has newly performed rendering.

[0057] Difference information 701 is an example of a difference region when the Web content rendering unit 511 renders the result 602 from the result 601 .

[0058] Difference information 702 is an example of a difference region when the Web content rendering unit 511 performs rendering from the result 601 to the result 602 and from the result 602 to the result 603.

[0059] Difference information 703 is an example of a difference region when the Web content rendering unit 511 performs rendering from the result 601 to the result 602, from the result 602 to the result 603, and from the result 603 to the result 604.

[0060] 8 is a diagram showing an example of image information generated by the maximum rectangle method from the difference information stored in the difference information storage unit 503 in this embodiment. The dotted lines indicate rectangular areas.

[0061] Image information 801 is an example of image information generated from difference information 701 when the result of rendering by Web content rendering unit 511 is result 602 .

[0062] Image information 802 is an example of image information generated from difference information 702 when the result of rendering by Web content rendering unit 511 is result 603 .

[0063] Image information 803 is an example of image information generated from difference information 703 when the result of rendering by Web content rendering unit 511 is result 604 .

[0064] FIG. 9 is a flowchart illustrating the flow of processing performed by the communication device 101 in this embodiment.

[0065] This flowchart starts when the communication device 101 starts an application in the application unit 306 and connects to the image generation device 103 via a network.

[0066] In step S901, the operation detection unit 303 determines whether it is an opportunity to transmit operation information. In this embodiment, the opportunity to transmit operation information is when the operation detection unit 303 detects an operation corresponding to a Web operation. For example, the opportunity to transmit operation information is when a click, touch, drag operation, or the like is detected.

[0067] If it is an opportunity to transmit operation information (Yes in S901), the process proceeds to step S902, and if it is not an opportunity to transmit operation information (No in S901), the process proceeds to step S904.

[0068] In step S902, the communication unit 301 transmits to the image generation device 103 the operation information of the Web page detected by the operation detection unit 303 in step S901.

[0069] In step S903, the communication unit 301 receives the operation information result from the image generation device 103, and the response analysis unit 305 analyzes the contents of the operation information result. The operation information result is information on the result of the operation information transmitted in step S902. This makes it possible to analyze whether the transmitted operation information was correctly processed by the image generation device 103.

[0070] In step S904, the communication unit 301 determines whether it is time to send an image information request. In this embodiment, the time to send an image information request is when a certain amount of time has passed since the screen was displayed in step S907 and the screen can be displayed in step S907. Whether the screen can be displayed in step S907 can be determined, for example, by determining whether the utilization rate of the CPU or memory of the communication device 101 is below a threshold.

[0071] If it is an opportunity to send an image information request (Yes in S904), the process proceeds to step S905, and if it is not an opportunity to send an image information request (No in S904), the process proceeds to step S908.

[0072] In step S905, the communication unit 301 transmits an image information request to the image generation device 103.

[0073] In step S906, the communication unit 301 receives the image information from the image generating device 103 and stores it in the storage unit 302.

[0074] In step S907, the screen generation unit 304 generates a screen from the image information stored in the storage unit 302, and controls the screen to be displayed on the graphical interface 204.

[0075] In step S908, the application unit 306 determines whether the application has ended. If the application has ended (Yes in S908), the process in Fig. 9 ends, and if the application has not ended (No in S908), the process proceeds to step S901.

[0076] In the process of FIG. 9, the determinations in steps S901, S904, and S908 are repeated until the result in step S908 is Yes.

[0077] 10 is a flowchart illustrating the processing flow of the image generation device 103 in this embodiment. This flowchart begins when the image generation device 103 launches a browser application. The size of the result of rendering by the Web content rendering unit 511 is assumed to be (x, y) = (800, 480). The processing method of the image generation device 103 will be described below.

[0078] In step S1001, the request determination unit 505 determines whether the communication unit 501 has received web page operation information from the communication device 101. If web page operation information has been received (Yes in S1001), the process proceeds to step S1002, and if web page operation information has not been received (No in S1001), the process proceeds to step S1004.

[0079] In step S1002, Web content analysis unit 510 reflects the operation information based on the operation information received in step S1001. In this embodiment, upon receiving the operation information, Web content rendering unit 511 starts rendering Web page results 602 to 604. Web content rendering unit 511 functions as a rendering unit, first rendering result 602, and then rendering results 603 and 604 in order over time.

[0080] In step S1003, the communication unit 501 transmits the operation information result to the communication apparatus 101. In this embodiment, the operation information result is information indicating whether or not the loading of the web page information has been successful.

[0081] In step S1004, rendering determination unit 504 determines whether rendering has been performed by Web content rendering unit 511. If rendering has been performed (Yes in S1004), the process proceeds to step S1005, and if rendering has not been performed (No in S1004), the process proceeds to step S1006.

[0082] In step S1005, difference information calculation unit 507 acquires the rendering result from Web content rendering unit 511, and stores the image information of the rendering result in storage unit 502. Thereafter, difference information calculation unit 507 calculates the difference region between the image information of the current rendering result and the image information of the previously stored rendering result, and stores the difference information including the difference region in difference information storage unit 503.

[0083] The difference information calculation unit 507 functions as an acquisition unit and acquires a difference region between the image information resulting from the current rendering in step S1002 and the image information resulting from the previous rendering. There may be one or more difference regions.

[0084] At this time, if the newly acquired differential region is included in an already stored differential region, the differential region in the differential information storage unit 503 is not updated. Furthermore, if the newly acquired differential region includes an already stored differential region, the included region is deleted, and the newly acquired differential region is stored in the differential information storage unit 503. This makes it possible to prevent duplicate differential regions from being stored.

[0085] In step S1006, the communication unit 501 determines whether or not an image information request has been received from the communication device 101. If an image information request has been received (Yes in S1006), the process proceeds to step S1007, and if an image information request has not been received (No in S1006), the process proceeds to step S1009.

[0086] In step S1007, the image information generation unit 508 generates image information from the difference information in the difference information storage unit 503. The generated image information is generated from one rectangular area that includes all of the difference areas included in the difference information. In this embodiment, this image information generation method is the maximum rectangle method.

[0087] The image information generation unit 508 uses the maximum rectangle method to generate image information of one rectangular area that includes all of the differential areas of step S1005. If multiple differential areas are acquired in step S1005, the image information generation unit 508 generates image information of one rectangular area that includes the multiple differential areas.

[0088] After generating the image information, image information generation unit 508 deletes the difference information from difference information storage unit 503. In the maximum rectangle method, for example, if the rendering result from Web content rendering unit 511 is result 604 and the state of the difference information in difference information storage unit 503 is difference information 703, image information 803 is generated.

[0089] In step S1008, the communication unit 501 functions as a transmission unit and transmits the image information generated in step S1007 to the communication device 101.

[0090] In step S1009, the browser application unit 506 determines whether the browser application has been terminated. If the browser application has been terminated (Yes in S1009), the process in FIG. 10 is terminated. If the browser application has not been terminated (No in S1009), the process proceeds to step S1001.

[0091] In the process of FIG. 10, the determinations in steps S1001, S1004, S1006, and S1009 are repeated until the result in step S1009 is Yes.

[0092] In this embodiment, rendering of the result 602 is performed in step S1002, and if the result is Yes in step S1004, the difference information calculation unit 507 acquires the difference region of the difference information 701 in step S1005. At this time, the information that can be acquired in step S1005 is (x, y)=(50 to 200, 300 to 400), and the state of the difference information stored in the difference information storage unit 503 becomes the difference information 701.

[0093] If the rendering of result 603 is performed in step S1002 as time passes, the information that can be obtained in step S1005 is (x, y) = (320 to 470, 300 to 400), and the state of the difference information stored in difference information storage unit 503 becomes difference information 702. At this time, if an image information request is received in step S1006, image information 802 is generated in step S1007.

[0094] According to this embodiment, the image generating device 103 stores the differential region as differential information and generates image information from the stored differential information, thereby transmitting only the differential information and reducing the amount of communication traffic. Furthermore, reducing the amount of image information to be transmitted not only reduces the processing load of the communication device 101 in step S907, but also improves the screen display speed.

[0095] According to this embodiment, the image generation device 103 receives the image information request in step S1006, thereby being able to know the timing when the communication device 101 can display the image information. Furthermore, the image generation device 103 transmits image information in step S1008 only if it receives the image information request in step S1006. This makes it possible to avoid receiving image information from the image generation device 103 when the communication device 101 is in a state where it is unable to display the screen in step S907, and to prevent a discrepancy between the rendering result of the image generation device 103 and the screen displayed by the communication device 101.

[0096] According to this embodiment, the image information generated in step S1007 consists of one rectangular area. Therefore, even if the time required for display in step S907 by the communication device 101 increases depending on the number of pieces of image information rather than the size of the image information, it is possible to display the image information faster than when the image information is divided into multiple rectangular areas and transmitted.

[0097] In this embodiment, the difference information is stored in the difference information storage unit 503 until the image information is generated in step S1007. This makes it possible to prevent differences from occurring between the image information of the communication device 101 and the image generation device 103.

[0098] (Second embodiment) The second embodiment is an embodiment in which the image generating device 103 in the first embodiment has a plurality of image information generating methods, and changes the image information generating method depending on the difference information in the difference information storage unit 503 .

[0099] The system configuration diagram, hardware configuration and functional configuration of the communication device 101 in the second embodiment are the same as those in Figs. 1 to 3, respectively. The hardware configuration and functional configuration diagram of the image generation device 103 are the same as those in Figs. 4 and 5, respectively. The processing flow of the communication device 101 is the same as that in Fig. 9.

[0100] FIG. 11 is a diagram showing an example of the result of rendering by the Web content rendering unit 511. As shown in FIG.

[0101] Result 1101 is an example of the result of the communication unit 501 receiving operation information from the communication device 101 and then performing the next rendering after the Web content rendering unit 511 has rendered the result 601 .

[0102] Result 1102 is an example of the result of the next rendering performed after the Web content rendering unit 511 has rendered result 1101.

[0103] Result 1103 is an example of the result of the next rendering performed after the Web content rendering unit 511 has rendered result 1102.

[0104] Result 1104 is an example of the result of the next rendering performed by communication unit 501 after Web content rendering unit 511 has rendered result 601 and communication unit 501 has received operation information from communication apparatus 101 .

[0105] Result 1105 is an example of the result of the next rendering performed over time after the Web content rendering unit 511 performed rendering of result 1104.

[0106] 12 is a diagram showing an example of difference information stored in the difference information storage unit 503 of this embodiment. Difference information 1201 to 1204 indicate the state of difference information stored in the difference information storage unit 503.

[0107] The difference information 1201 to 1204 includes a difference area, and the difference area indicates an area where the Web content rendering unit 511 has performed rendering.

[0108] Difference information 1201 is an example of a difference region when the Web content rendering unit 511 renders the result 1101 from the result 601 .

[0109] Difference information 1202 is an example of a difference region when the Web content rendering unit 511 performs rendering from the result 601 to the result 1101 and from the result 1101 to the result 1102.

[0110] Difference information 1203 is an example of a difference region when the Web content rendering unit 511 performs rendering from the result 601 to the result 1101, from the result 1101 to the result 1102, and from the result 1102 to the result 1103.

[0111] Difference information 1204 is an example of a difference region between when the Web content rendering unit 511 renders the result 1104 from the result 601 and when the Web content rendering unit 511 renders the result 1105 from the result 1104 .

[0112] 13 is a diagram showing an example of image information generated by the maximum rectangle method from difference information stored in the difference information storage unit 503 in this embodiment. The dotted lines indicate rectangular areas.

[0113] Image information 1301 is an example of image information generated from difference information 1201 when the result of rendering by Web content rendering unit 511 is result 1101 .

[0114] Image information 1302 is an example of image information generated from difference information 1202 when the result of rendering by Web content rendering unit 511 is result 1102 .

[0115] Image information 1303 is an example of image information generated from difference information 1203 when the result of rendering by Web content rendering unit 511 is result 1103 .

[0116] FIG. 14 is a diagram showing an example of image information generated by the differential region method from the differential information stored in the differential information storage unit 503 in this embodiment.

[0117] Image information 1401 is an example of image information generated from information on the differential region (x, y)=(50 to 200, 300 to 400) when the result of rendering by Web content rendering unit 511 is result 602.

[0118] Image information 1402 is an example of image information generated from information on the differential region (x, y)=(320 to 470, 300 to 400) when the result of rendering by Web content rendering unit 511 is result 603.

[0119] Image information 1403 is an example of image information generated from information on the differential region (x, y)=(590 to 740, 300 to 400) when the result of rendering by Web content rendering unit 511 is result 604.

[0120] 15 is a diagram showing an example of screen division using the cell comparison method in this embodiment, and an example of image information generated using the cell comparison method from difference information stored in the difference information storage unit 503. The dotted lines indicate rectangular areas.

[0121] Screen 1501 is an example of a screen in which the screen displaying the image when Web content rendering unit 511 renders result 604 is divided equally into six rectangular areas.

[0122] Image information 1502 is an example of image information generated from information on the differential region (x, y)=(50 to 200, 300 to 400) when the result of rendering by Web content rendering unit 511 is result 602.

[0123] Image information 1503 is an example of image information generated from information on the differential region (x, y)=(320 to 470, 300 to 400) when the result of rendering by Web content rendering unit 511 is result 603.

[0124] Image information 1504 is an example of image information generated from information on the differential region (x, y)=(590 to 740, 300 to 400) when the result of rendering by Web content rendering unit 511 is result 604.

[0125] 16 is a flowchart illustrating the processing flow of the image generation device 103 in this embodiment. This flowchart begins when the image generation device 103 starts a browser application. The size of the result of rendering by the Web content rendering unit 511 is assumed to be (x, y) = (800, 480).

[0126] The image generating device 103 performs the processes of steps S1001 to S1009. Steps S1001 to S1009 in Fig. 16 are the same processes as steps S1001 to S1009 in Fig. 10, respectively.

[0127] In step S1006, if an image information request is received (Yes in S1006), the process proceeds to step S1601.

[0128] In step S1601, image information generation means determination unit 509 determines whether there is one differential region in the differential information stored in difference information storage unit 503. If there is one differential region (Yes in S1601), the process proceeds to step S1602, and if there is not one differential region (No in S1602), the process proceeds to step S1603.

[0129] In step S1602, the image information generation unit 508 generates image information from the difference information in the difference information storage unit 503. The generated image information is generated from each difference region included in the difference information. In this embodiment, this image information generation method is the difference region method.

[0130] If multiple differential regions are acquired in step S1005, the image information generating unit 508 generates image information of multiple rectangular regions from the multiple differential regions using the differential region method.

[0131] After generating the image information, image information generation unit 508 deletes the difference information from difference information storage unit 503. In the difference region method, for example, if the rendering result from Web content rendering unit 511 is result 604 and the state of the difference information in difference information storage unit 503 is difference information 703, image information generation unit 508 generates image information 1401, 1402, and 1403. Then, the process proceeds to step S1008.

[0132] In step S1603, the image information generation means determination unit 509 calculates the amount of difference for generating image information from the difference information stored in the difference information storage unit 503 using the difference area method, the maximum rectangle method, and the cell comparison method.

[0133] In step S1604, the image information generation means determination unit 509 determines the result of the calculation in step S1603. If the difference amount in the maximum rectangle method is the smallest (Yes in S1604), proceed to step S1607; otherwise (No in S1604), proceed to step S1605.

[0134] In this embodiment, in step S1007, for example, if the rendering result of the Web content drawing unit 511 is result 1103 and the state of the difference information in the difference information storage unit 503 is difference information 1203, the image information generation means determination unit 509 generates image information 1303.

[0135] In step S1605, the image information generation means determination unit 509 determines the result of the calculation in step S1603. If the difference amount in the differential domain method is the smallest (Yes in S1605), the process proceeds to step S1602; otherwise (No in S1605), the process proceeds to step S1606.

[0136] In step S1606, image information generation unit 508 generates image information from the difference information in difference information storage unit 503. The image information to be generated is generated by dividing screen 1501 into multiple rectangular areas in advance, and if any of the divided rectangular areas includes a difference area included in the difference information, image information for that rectangular area is generated. In this embodiment, this image information generation method is a cell comparison method. Image information generation unit 508 uses the cell comparison method to generate image information for a rectangular area that includes the difference area of ​​step S1005, out of the multiple rectangular areas obtained by dividing screen 1501.

[0137] In this embodiment, screen 1501 is divided equally into six rectangular regions, but this is not limiting and the number of divisions may be more than six. Screen 1501 is shown as an example of screen 1501 divided equally into six rectangular regions. After generating image information, image information generation unit 508 deletes the difference information stored in difference information storage unit 503. In the cell comparison method, for example, if the rendering result of Web content rendering unit 511 is result 604 and the state of the difference information stored in difference information storage unit 503 is difference information 703, image information generation unit 508 generates image information 1502, 1503, and 1504. Then, the process proceeds to step S1008.

[0138] In step S1008, the communication unit 501 transmits the image information generated in step S1007, S1602, or S1606 to the communication device 101. As described above, the communication unit 501 transmits image information generated by a method that can generate the smallest amount of image information among the image information generated by the maximum rectangle method, the differential area method, and the cell comparison method.

[0139] According to this embodiment, when rendering occurs for the same location, such as during video playback or animation processing, image information can be generated from difference information using a method that can further reduce the amount of image information. For example, if the Web content rendering unit 511 performs rendering in the order of results 601, 1104, and 1105 in the processing of step S1002, the difference information will be in the state of difference information 1204 when result 1105 is rendered. This is because it is possible to prevent duplicate difference areas from being stored in step S1005. In other words, even when rendering occurs multiple times for the same location, such as during motion playback or animation processing, only one difference area is stored in step S1005. In this case, the amount of image information can be reduced by not generating images using the cell comparison method.

[0140] In this embodiment, in step S1603, the difference amount is calculated using the difference area method, maximum rectangle method, and cell comparison method. With the difference area method, the range of each difference area in difference information storage unit 503 is calculated, and the sum of these calculations is used as the difference amount. With the maximum rectangle method, the range of one rectangular area that includes all of the difference areas in difference information storage unit 503 is used as the difference amount. With the cell comparison method, the range of each divided screen area that includes the difference areas in difference information storage unit 503 is calculated, and the sum of these calculations is used as the difference amount.

[0141] Thereafter, the method with the smallest difference is selected in the determinations of steps S1604 and S1605, and image information is generated using the selected method. As a result, image information can be generated using the method with the smallest difference, and as a result, image information can be generated using a method that can further reduce the amount of image information.

[0142] In this embodiment, the difference amount is calculated in step S1603, but this is not limited to this. Image information may be generated using the difference area method, maximum rectangle method, or cell comparison method, and the image information generated using the method that produces the smallest image information may be transmitted in step S1008.

[0143] In this embodiment, the difference information is stored in the difference information storage unit 503 until the image information is generated in steps S1007, S1602, and S1606. This makes it possible to prevent differences from occurring between the image information of the communication device 101 and the image generation device 103.

[0144] (Third embodiment) The third embodiment is an embodiment in which the image generating device 103 receives an upper limit on the number of pieces of image information from the communication device 101 in the second embodiment, and changes the image information generating method based on the upper limit and the difference information in the difference information storage unit 503.

[0145] The system configuration diagram, hardware configuration and functional configuration of the communication device 101 in the third embodiment are the same as those in Figs. 1 to 3, respectively. The hardware configuration and functional configuration diagram of the image generation device 103 are the same as those in Figs. 4 and 5, respectively. The processing flow of the communication device 101 is the same as that in Fig. 9.

[0146] 17 is a flowchart illustrating the processing flow of the image generation device 103 in the third embodiment. This flowchart starts when the image generation device 103 starts a browser application. The size of the result of rendering by the Web content rendering unit 511 is assumed to be (x, y)=(800, 480).

[0147] Image generation device 103 performs the processes of steps S1001 to S1009 and S1601 to S1606. Steps S1001 to S1009 in Fig. 17 are the same processes as steps S1001 to S1009, respectively, in Fig. 10. Steps S1601 to S1606 in Fig. 17 are the same processes as steps S1601 to S1606, respectively, in Fig. 16.

[0148] After step S1003, in step S1701, the communication unit 501 determines whether or not an upper limit on the number of pieces of image information has been received from the communication device 101. If an upper limit on the number of pieces of image information has been received (Yes in S1701), the process proceeds to step S1702, and if an upper limit on the number of pieces of image information has not been received (No in S1702), the process proceeds to step S1004.

[0149] In step S1702, the communication unit 501 updates the upper limit of the number of pieces of image information stored in the storage unit 502 to the upper limit received in step S1701. Then, the process proceeds to step S1004.

[0150] In step S1605, if the difference amount in the differential domain method is the smallest (Yes in S1605), the process proceeds to step S1703; otherwise (No in S1605), the process proceeds to step S1704.

[0151] In step S1703, the image information generation means determination unit 509 determines, from the amount of difference in the differential domain method calculated in step S1603, whether the number of pieces of image information generated in step S1602 is equal to or less than the upper limit of the number of pieces of image information stored in the storage unit 502. If the number of pieces of image information is equal to or less than the upper limit (Yes in S1703), the process proceeds to step S1602, and if the number of pieces of image information is not equal to or less than the upper limit (No in S1703), the process proceeds to step S1704.

[0152] In step S1704, image information generation means determination unit 509 determines, from the difference amount in the cell comparison method calculated in step S1603, whether the number of pieces of image information to be generated in step S1606 is equal to or less than the upper limit of the number of pieces of image information stored in storage unit 502. If it is equal to or less than the upper limit of the number of pieces of image information (Yes in S1704), the process proceeds to step S1606, and if it is not equal to or less than the upper limit of the number of pieces of image information (No in S1704), the process proceeds to step S1007.

[0153] As described above, if the amount of image information generated by the maximum rectangle method is the smallest among the image information generated by the maximum rectangle method, the differential region method, and the cell comparison method, the process proceeds from step S1604 to step S1008 via step S1007. In this case, the communication unit 501 transmits the image information generated by the maximum rectangle method in step S1007 to the communication device 101.

[0154] If the amount of image information generated by the differential domain method is the smallest among the image information generated by the maximum rectangle method, the differential domain method, and the cell comparison method, and the amount of image information generated by the differential domain method is equal to or less than the upper limit, the process proceeds from step S1605 to step S1703 and step S1602, and then to step S1008. In this case, the communication unit 501 transmits the image information generated by the differential domain method in step S1602 to the communication device 101.

[0155] If the amount of image information generated by the cell comparison method is the smallest among the image information generated by the maximum rectangle method, the differential region method, and the cell comparison method, and the amount of image information generated by the cell comparison method is equal to or less than the upper limit, the process proceeds to S1008 via steps S1605 to S1704 and S1606. In this case, the communication unit 501 transmits the image information generated by the cell comparison method in step S1606 to the communication device 101.

[0156] If the amount of image information generated by the differential domain method is the smallest among the image information generated by the maximum rectangle method, the differential domain method, and the cell comparison method, and if the amount of image information generated by the differential domain method is not less than the upper limit, and if the amount of image information generated by the cell comparison method is less than the upper limit, the process proceeds from step S1605 to S1008 via S1703, S1704, and S1606. In this case, the communication unit 501 transmits the image information generated by the cell comparison method in step S1606 to the communication device 101.

[0157] If the amount of image information generated by the differential domain method is the smallest among the image information generated by the maximum rectangle method, the differential domain method, and the cell comparison method, and if the amount of image information generated by the differential domain method is not less than the upper limit, and the amount of image information generated by the cell comparison method is not less than the upper limit, the process proceeds from step S1605 to step S1703, S1704, and S1007, and then to step S1008. In this case, the communication unit 501 transmits the image information generated by the maximum rectangle method in step S1007 to the communication device 101.

[0158] If the amount of image information generated by the cell comparison method is the smallest among the image information generated by the maximum rectangle method, the differential region method, and the cell comparison method, and the amount of image information generated by the cell comparison method is not equal to or less than the upper limit, the process proceeds from step S1605 to step S1704 and step S1007, and then to step S1008. In this case, the communication unit 501 transmits the image information generated by the maximum rectangle method in step S1007 to the communication device 101.

[0159] According to this embodiment, the communication apparatus 101 transmits the upper limit of the number of pieces of image information, so that the number of pieces of image information transmitted in step S1008 can be made equal to or less than the upper limit of the number of pieces of image information requested by the communication apparatus 101.

[0160] Therefore, even if the time required for display in step S907 of the communication device 101 increases depending on the number of pieces of image information rather than the size of the image information, it is possible to suppress an increase in the time required for display in step S907. Furthermore, steps S1603, S1604, and S1605 enable image information to be generated using a method that minimizes the amount of difference, so that image information can be generated from difference information using a method that can further reduce the amount of image information.

[0161] According to the first to third embodiments, no difference occurs between the screen data of the communication device 101 and the image generation device 103, and the amount of screen data to be transmitted can be reduced.

[0162] (Other embodiments) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0163] It should be noted that the above-described embodiments merely illustrate specific examples of implementing the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these embodiments. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0164] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a rendering means for rendering the web page upon receiving the web page operation information; an acquisition means for acquiring a difference region between image information resulting from the current rendering by the rendering means and image information resulting from the previous rendering; a first generating means for generating image information of one rectangular area including the differential area when receiving an image information request; a transmitting means for transmitting image information of one rectangular area generated by the first generating means; An image generating device comprising: (Configuration 2) the acquiring means acquires a plurality of differential regions; 2. The image generating device according to configuration 1, wherein the first generating means generates image information of one rectangular area including the plurality of differential areas. (Configuration 3) The method further includes a second generating means for generating image information of a plurality of rectangular regions from the plurality of differential regions, 3. The image generating device according to configuration 2, wherein the transmitting means transmits the image information generated by the first generating means or the second generating means. (Configuration 4) The method further includes a third generation means for generating image information of a rectangular area including the difference area among a plurality of rectangular areas obtained by dividing the screen, 3. The image generating device according to configuration 2, wherein the transmitting means transmits the image information generated by the first generating means or the third generating means. (Configuration 5) The method further includes a third generation means for generating image information of a rectangular area including the difference area among a plurality of rectangular areas obtained by dividing the screen, 4. The image generating device according to configuration 3, wherein the transmitting means transmits the image information generated by the first generating means, the second generating means, or the third generating means. (Configuration 6) The image generating device according to configuration 5, wherein the transmitting means transmits image information generated by a generating means capable of generating the smallest amount of image information among the image information generated by the first to third generating means. (Configuration 7) The transmitting means If the amount of image information generated by the first generation means is the smallest among the amount of image information generated by the first to third generation means, transmit the image information generated by the first generation means; If the amount of image information generated by the second generating means is the smallest among the image information generated by the first to third generating means and the number of image information generated by the second generating means is equal to or less than an upper limit, transmit the image information generated by the second generating means; The image generating device according to configuration 6, characterized in that when the amount of image information generated by the third generating means is the smallest among the image information generated by the first to third generating means and the number of image information generated by the third generating means is equal to or less than an upper limit, the image generating device transmits the image information generated by the third generating means. (Configuration 8) The transmitting means If the amount of image information generated by the second generating means is the smallest among the image information generated by the first to third generating means, and the amount of image information generated by the second generating means is not less than the upper limit, and the amount of image information generated by the third generating means is less than the upper limit, transmit the image information generated by the third generating means; The image generating device described in configuration 7 is characterized in that, when the amount of image information generated by the second generating means is the smallest among the image information generated by the first to third generating means, the amount of image information generated by the second generating means is not less than an upper limit, and the amount of image information generated by the third generating means is not less than an upper limit, the image generating device transmits the image information generated by the first generating means. (Configuration 9) The image generating device described in configuration 8 is characterized in that the transmitting means transmits the image information generated by the first generating means when the amount of image information generated by the third generating means is the smallest among the image information generated by the first to third generating means and the amount of image information generated by the third generating means is not below an upper limit. (Configuration 10) The image generating device according to any one of configurations 1 to 9, a communication device; An information processing system characterized in that the communication device sends operation information of the web page and the image information request to the image generation device, receives the image information, and controls the image generation device to display a screen based on the image information. (Method 1) a rendering step of rendering the web page upon receiving the web page operation information; an acquisition step of acquiring a difference region between image information of the current rendering result in the rendering step and image information of the previous rendering result; a first generation step of generating image information of one rectangular area including the difference area when an image information request is received; a transmission step of transmitting image information of one rectangular area generated by the first generation step; A processing method for an image generating device, comprising: (Program 1) A program for causing a computer to function as the image generating device according to any one of the first to ninth aspects. [Explanation of symbols]

[0165] 501 communication unit, 502 storage unit, 503 difference information storage unit, 504 rendering determination unit, 505 request determination unit, 506 browser application unit, 507 difference information calculation unit, 508 image information generation unit, 509 image information generation means determination unit

Claims

1. a rendering means for rendering the web page upon receiving operation information of the web page; an acquisition means for acquiring a difference region between image information resulting from the current rendering by the rendering means and image information resulting from the previous rendering; a first generating means for generating image information of one rectangular area including the differential area when receiving an image information request; a transmitting means for transmitting image information of one rectangular area generated by the first generating means; An image generating device comprising:

2. the acquiring means acquires a plurality of differential regions; 2. The image generating device according to claim 1, wherein the first generating means generates image information of one rectangular area including the plurality of differential areas.

3. a second generating means for generating image information of a plurality of rectangular regions from the plurality of differential regions; 3. The image generating device according to claim 2, wherein the transmitting means transmits the image information generated by the first generating means or the second generating means.

4. a third generation means for generating image information of a rectangular area including the difference area among a plurality of rectangular areas obtained by dividing the screen; 3. The image generating device according to claim 2, wherein the transmitting means transmits the image information generated by the first generating means or the third generating means.

5. a third generation means for generating image information of a rectangular area including the difference area among a plurality of rectangular areas obtained by dividing the screen; 4. The image generating device according to claim 3, wherein the transmitting means transmits the image information generated by the first generating means, the second generating means, or the third generating means.

6. The image generating device according to claim 5, characterized in that the transmitting means transmits image information generated by a generating means capable of generating the smallest amount of image information among the image information generated by the first to third generating means.

7. The transmitting means If the amount of image information generated by the first generation means is the smallest among the amount of image information generated by the first to third generation means, transmit the image information generated by the first generation means; If the amount of image information generated by the second generating means is the smallest among the image information generated by the first to third generating means and the number of image information generated by the second generating means is equal to or less than an upper limit, transmit the image information generated by the second generating means; The image generating device according to claim 6, characterized in that, when the amount of image information generated by the third generating means is the smallest among the image information generated by the first to third generating means and the number of image information generated by the third generating means is equal to or less than an upper limit, the image information generated by the third generating means is transmitted.

8. The transmitting means If the amount of image information generated by the second generating means is the smallest among the image information generated by the first to third generating means, and the amount of image information generated by the second generating means is not less than the upper limit, and the amount of image information generated by the third generating means is less than the upper limit, transmit the image information generated by the third generating means; The image generating device described in claim 7, characterized in that if the amount of image information generated by the second generating means is the smallest among the image information generated by the first to third generating means, and the number of image information generated by the second generating means is not below an upper limit, and the number of image information generated by the third generating means is not below an upper limit, the image generating device transmits the image information generated by the first generating means.

9. The image generating device described in claim 8, characterized in that the transmitting means transmits the image information generated by the first generating means when the amount of image information generated by the third generating means is the smallest among the image information generated by the first to third generating means and the number of image information generated by the third generating means is not below an upper limit.

10. An image generating device according to any one of claims 1 to 9; a communication device; An information processing system characterized in that the communication device transmits operation information of the web page and the image information request to the image generation device, receives the image information, and controls the image generation device to display a screen based on the image information.

11. a rendering step of rendering the web page when receiving the web page operation information; an acquisition step of acquiring a difference region between image information of the current rendering result in the rendering step and image information of the previous rendering result; a first generation step of generating image information of one rectangular area including the difference area when an image information request is received; a transmitting step of transmitting image information of one rectangular area generated by the first generating step; A processing method for an image generating device, comprising:

12. A program for causing a computer to function as the image generating device according to any one of claims 1 to 9.

Images