Information processing system, information processing device, terminal device, control method and program

Abstract

This provides technology for displaying images appropriately. [Solution] The information processing system comprises an information processing device and a terminal device. The information processing device selects a first element contained in a web page and generates a first image, which is the result of drawing only the first element within a first area of ​​the web page, and a second image, which includes the result of drawing a second element other than the first element within a second area that is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device. The information processing device transmits the first image and the second image to the terminal device. Based on the occurrence of a scroll operation on the terminal device, the terminal device extracts a third image from the second area from the first image and displays the third image. Based on the occurrence of a scroll operation on the terminal device, the terminal device receives a second image corresponding to the change in the display range from the information processing device and displays the second image corresponding to the change in the display range.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing apparatus, a terminal device, a control method, and a program.

Background Art

[0002] Conventionally, communication terminals such as image processing apparatuses (information processing apparatuses) equipped with a Web browser (hereinafter referred to as a browser) and having a function of browsing Web pages (hereinafter also simply referred to as pages) on the browser have been known. With the improvement of the browser function, not only simple information browsing but also so-called Web applications that act on data on the Internet and execute various procedures can be used by communication terminals.

[0003] Recently, a mechanism called a cloud browser that uses an image generation server that generates a rendering result of a Web page on a cloud server has been studied. Patent Document 1 discloses a system in which a Web page is rendered (drawn) in a virtual machine on a network different from a communication terminal, and the rendering result is displayed on the communication terminal.

[0004] In a cloud browser, the rendering result of a Web page is cut out to the display size of a communication terminal and transmitted. Then, every time the display range within the Web page is changed by a scroll operation such as a swipe on the display, the corresponding range is cut out and transmitted. By doing so, it becomes possible to view large Web pages even on communication terminals with little memory.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] As mentioned above, when providing scrolling functionality for web pages on a communication device, updating the display in accordance with finger movement can provide the same usability as a browser on a typical mobile device. However, since communication occurs when obtaining the image of the scrolled area, frequent updates increase the load on the device and communication network. Furthermore, there are limitations to the update frequency due to communication latency. In addition, since the display will be blank until the image is received, subsequent operations such as clicking links or scrolling further cannot be performed during that time, reducing browsing efficiency.

[0007] To reduce the number of communications during scrolling, the above issues can be mitigated by pre-loading images of the scrolled area from the cloud or saving images of previously viewed areas to the cache on the communication terminal. However, this method consumes a large amount of memory on the communication terminal, so there are limits to its effectiveness in reducing communication traffic.

[0008] One aspect of this disclosure, in view of the above, aims to provide a technology for appropriately displaying images. [Means for solving the problem]

[0009] An information processing system according to one aspect of the present disclosure comprises an information processing device and a terminal device, wherein the information processing device comprises: a selection means for selecting a first element included in a web page; a generation means for generating a first image which is the result of drawing only the first element in a first area of ​​the web page and a second image which includes the result of drawing a second element other than the first element in a second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device; and a transmission means for transmitting the first image and the second image to the terminal device, wherein the terminal device comprises: a receiving means for receiving the first image and the second image from the information processing device; and a display control means for cutting out a third image in the second area from the first image and displaying the third image based on the occurrence of a scroll operation on the terminal device, wherein the receiving means receives the second image corresponding to the change in the display range from the information processing device based on the occurrence of a scroll operation on the terminal device, and the display control means displays the second image corresponding to the change in the display range. [Effects of the Invention]

[0010] According to one aspect of this disclosure, images can be displayed appropriately. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example of the hardware configuration of a communication terminal and an image generation server according to the first embodiment. [Figure 2] This figure shows an example configuration of an information processing system according to the first embodiment. [Figure 3] This is a flowchart showing an example of processing by the image generation server according to the first embodiment. [Figure 4] This is a flowchart showing an example of processing at a communication terminal according to the first embodiment. [Figure 5] This is a flowchart showing a detailed example of the process in step S305 in Figure 3, according to the first embodiment. [Figure 6]It is a schematic diagram showing a page image and a display range in FIG. 3 according to the first embodiment. [Figure 7] It is a diagram showing an example of a table for setting priorities for elements according to the second embodiment. [Figure 8] It is a flowchart showing an example of processing of an image generation server according to the third embodiment. [Figure 9] It is a flowchart showing an example of processing of a communication terminal according to the third embodiment. [Figure 10] It is a diagram showing an example of a page image generated by an image generation server according to the first embodiment. [Figure 11] It is a diagram showing an example of an image displayed by a communication terminal according to the first embodiment. [Figure 12] It is a flowchart showing a detailed processing example of step S305 in FIG. 3 according to the fourth embodiment. [Figure 13] It is a flowchart showing a detailed processing example of step S405 in FIG. 4 according to the fourth embodiment.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the content described in the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the present disclosure, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0013] <The First Embodiment> First, the hardware configurations of a communication terminal (an example of a terminal device) and an image generation server (an example of an information processing device), which are components of the information processing system according to the first embodiment, will be described with reference to FIG. 1.

[0014] FIG. 1(a) is a block diagram showing an example of the hardware configuration of a communication terminal according to the first embodiment.

[0015] The communication terminal 100 includes a CPU 101, a ROM 102, a RAM 103, a display unit 104, an input unit 105, and a communication unit 106. The CPU 101, the ROM 102, the RAM 103, the display unit 104, the input unit 105, and the communication unit 106 are communicably connected to each other via a system bus 107.

[0016] The CPU (Central Processing Unit) 101 controls the entire communication terminal 100. For example, the CPU 101 executes each process of the communication terminal 100 by reading and executing programs stored in the ROM 102, the RAM 103, etc.

[0017] The ROM (Read Only Memory) 102 stores programs, parameters, etc. that do not require modification. Examples of the ROM 102 include a mask ROM, a flash memory, a CD (Compact Disk)-ROM read by an optical drive, etc.

[0018] The RAM (Random Access Memory) 103 temporarily stores programs, data, etc. supplied from external devices, network destinations, etc.

[0019] The display unit 104 is a graphical user interface that displays data held by the communication terminal 100, data supplied to the communication terminal 100, etc. Examples of the display unit 104 include a liquid crystal display, etc.

[0020] The input unit 105 is a device that receives operations from the user. Examples of the input unit 105 include a pointing device such as a mouse, a keyboard device, etc. The display unit 104 such as a liquid crystal display may include the function of the input unit 105 as a touch panel.

[0021] The communication unit 106 is a network interface for connecting to a network (wired or wireless network). Examples of networks include the internet, intranets, and wireless communication networks.

[0022] Figure 1(b) is a block diagram showing an example of the hardware configuration of an image generation server according to the first embodiment.

[0023] The image generation server 110 comprises a CPU 111, ROM 112, RAM 113, and interface 114. The CPU 111, ROM 112, RAM 113, and interface 114 are interconnected via a system bus 115 so that they can communicate with each other.

[0024] The CPU 111 controls the entire image generation server 110. For example, the CPU 111 executes various processes of the image generation server 110 by reading and executing programs stored in ROM 112, RAM 113, etc.

[0025] ROM112 stores programs, parameters, etc., that do not require modification.

[0026] RAM113 temporarily stores programs, data, etc., supplied from external devices, network destinations, etc.

[0027] A hard disk, memory card, etc., which is virtually fixed and installed on the image generation server 110, may exist to perform some or all of the roles of ROM 112 or RAM 113. Alternatively, an external storage device, including a flexible disk (FD), optical disk such as a CD, magnetic card, optical card, IC (Integrated Circuit) card, memory card, etc., which can be virtually attached to and detached from the image generation server 110, may exist.

[0028] Interface 114 is an interface for inputting and outputting data between the image generation server 110 and external devices (for inputting operation instructions and data to the image generation server 110, and outputting processing results from the image generation server 110). Specifically, interface 114 includes a network interface for connecting to a network, enabling input and output by virtual terminals on the network. Interface 114 may also include a USB (Universal Serial Bus) interface for connecting to physical input devices such as a mouse and keyboard, and display devices such as an external display.

[0029] The image generation server 110 may be a so-called "physical server" having a CPU 111, ROM 112, RAM 113, and interface 114 in hardware. Alternatively, the image generation server 110 may be a so-called "virtual server" run on such a physical server by a program.

[0030] Next, the functional configuration for realizing a cloud browser in the information processing system according to this embodiment will be described with reference to Figure 2.

[0031] This information processing system includes a communication terminal 100 and a cloud 200 (located on the internet). An image generation server 110 resides within the cloud 200. The communication terminal 100 and image generation server 110 shown in Figure 2 correspond to the communication terminal 100 and image generation server 110 described with reference to Figure 1, respectively. In this information processing system, the network-connected communication terminal 100 and image generation server 110 work together to browse web pages on websites (not shown) on the internet.

[0032] The communication terminal 100 includes a display control unit 201 and a transmitting / receiving unit 202.

[0033] The display control unit 201 generates an image conversion request for the image generation server 110 to display (including updating) the web page on the display unit 104, based on the user's specified page and in-page operations. The display control unit 201 also decompresses and extracts the compressed image data obtained as a response to the image conversion request, and performs display control such as displaying the resulting image in accordance with user operations.

[0034] As described later, the display control unit 201, based on the occurrence of a scroll operation on the communication terminal 100, extracts an image from a second area, which is narrower than the first area, from the first image within the first area of ​​the web page, and displays the extracted image. The first image is the rendering result of only the first element included in the web page, and the first element may be, for example, a foreground element. The second area is the area corresponding to the display range of the communication terminal 100. Furthermore, as described later, the display control unit 201, based on the occurrence of no scroll operation on the communication terminal 100, displays a second image within the second area, which is received from the image generation server 110 and corresponds to the change in the display range of the communication terminal 100. The second image includes the rendering results of the second element other than the first element.

[0035] The transmitting / receiving unit 202 uses a communication protocol (communication means) such as HTTP (Hyper Text Transfer Protocol) to send an image creation request to the image generation server 110 and receives a response to the image creation request from the image generation server 110.

[0036] The image generation server 110 includes a browser unit 211, a transmission / reception unit 212, an element selection unit 213, and an image compression unit 214.

[0037] The browser unit 211 acquires a web page (via the transmission / reception unit 212) based on an image request from the communication terminal 100. The browser unit 211 also generates a page image by rendering the acquired web page data based on the drawing elements selected by the element selection unit 213.

[0038] The browser unit 211 generates a first image, which is the rendering result of only the first element within the first area of ​​the web page, as will be described later. The first element may be, for example, a foreground element. The browser unit 211 also generates a second image, which is within a second area that is narrower than the first area of ​​the web page, as will be described later. The second area corresponds to the display range of the communication terminal 100. The second image includes the rendering results of the second element other than the first element.

[0039] The transmitting / receiving unit 212 sends an HTTP request to the website when the browser unit 211 retrieves a web page, and receives the web page as a response. In addition, the transmitting / receiving unit 212 sends the page image generated by the browser unit 211 and compressed by the image compression unit 214 to the communication terminal 100 as a response to an image conversion request.

[0040] The element selection unit 213 selects the drawing elements when the browser unit 211 converts web page data into an image. The selection performed by the element selection unit 213 is based on the type of element, such as text, images, buttons, and tables, as well as the role of each element assigned by attributes such as links.

[0041] The image compression unit 214 applies image compression to the page image transmitted by the transmission / reception unit 212 in order to reduce the amount of data.

[0042] The operation of the image generation server 110 in this embodiment will be described with reference to Figure 3.

[0043] Figure 3 is a flowchart showing an example of processing by the image generation server according to the first embodiment.

[0044] In step S301, the image generation server 110 receives a request from the communication terminal 100 to convert a web page into an image.

[0045] The above image creation request includes the URL (Uniform Resource Locator) of the target web page and a specification of the display range of the page image. Often, the display unit 104 of the communication terminal 100 lacks sufficient size and resolution to display the entire web page in a visible manner. Therefore, when creating an image creation request, the communication terminal 100 specifies a display range representing a portion of the page image and displays the resulting partial image. Then, based on the user's operation to change the display range, the communication terminal 100 requests an image of the changed display range and updates the display using the resulting partial image.

[0046] The method for specifying the display range of page images will be explained with reference to Figure 6.

[0047] Figure 6 is a schematic diagram showing the page image and display range in Figure 3 according to the first embodiment.

[0048] Page image 601 is an image that renders the entire content of the web page requested by the browser unit 211. Page image 601 has a page size (Pw, Ph) determined by the number of pixels in the width and height. Pw represents the number of pixels in the width direction of page image 601 (page width), and Ph represents the number of pixels in the height direction of page image 601 (page height).

[0049] The display range 602 is the display range within the page image 601 when the communication terminal 100 displays a portion of the page image 601 on the display unit 104. The display range 602 consists of offset coordinates (Px, Py) from the page origin and the display size (W, H) of the communication terminal 100. W represents the display width, and H represents the display height.

[0050] In this embodiment, the image generation server 110 will create an image of the web page using the display width (W) = page width (Pw) specified by the communication terminal 100. Therefore, in the display range 602 actually specified by the communication terminal 100, the three values ​​Px (=0), W, and H will be fixed.

[0051] Returning to the explanation of Figure 3, in step S302, the image generation server 110 determines whether or not page data has already been obtained for the requested target web page URL. If the page data has already been obtained (YES in step S302), the image generation server 110 proceeds to step S307; if the page data has not already been obtained (NO in step S302), it proceeds to step S303.

[0052] Here, to determine whether page data has been acquired, the image generation server 110 can determine that page data has been acquired if the page data acquisition time associated with the target URL is recorded in step S304, which will be described later. Alternatively, the image generation server 110 may compare the last page data acquisition time with the current time and determine that page data has not been acquired if there is a difference of a certain amount or more. Or, the image generation server 110 may acquire the update date and time of the URL destination page and determine that page data has not been acquired if the update date and time are newer than the page data acquisition time. In this example, it is the first image generation request for the target URL, and the image generation server 110 proceeds to step S303, assuming that page data has not been acquired.

[0053] In step S303, the image generation server 110 obtains page data from the URL of the web page. The page data is generally written in HTML, but it may also include descriptions in other markup languages ​​such as SVG (Scalable Vector Graphics) or scripts such as JavaScript (registered trademark).

[0054] In step S304, the image generation server 110 records the current time as the time the page data for the target URL is acquired, associating it with the target URL.

[0055] In step S305, the image generation server 110 executes the process of generating a page image from the acquired page data. The details of the process in step S305 will be explained with reference to Figure 5.

[0056] Figure 5 is a flowchart showing a detailed example of the process in step S305 in Figure 3.

[0057] In step S501, the image generation server 110 analyzes the page data acquired in step S303 in Figure 3 and selects foreground elements to render the foreground.

[0058] In this embodiment, the foreground refers to pixels (or sets of pixels) that hold information of relatively high importance to the viewer within a page image obtained by visualizing a web page. Generally, in web pages that provide document information, character pixels (character pixels) are important foreground elements, but other pixels such as lines and table frames that indicate the document structure, and pixels such as input frames and button frames related to user operations also constitute the foreground. Therefore, the foreground (foreground element) may be one or more of characters, table frames, input frames, and button frames. Alternatively, depending on the web page, pixels that render photographs or illustrations may also be foreground elements. What is actually defined as the foreground and what elements are selected as foreground elements to render the foreground are predetermined in the element selection unit 213.

[0059] The following describes the process when, as an example, the element that renders the pixels of all text and button frames on a web page is designated as the foreground element. In this case, in step S501, all text nodes and button nodes included in the DOM (Document Object Model) tree of the target web page are selected as foreground elements.

[0060] In step S502, the image generation server 110 generates a foreground image. The foreground image is a page image in which only the foreground elements selected in step S501 are rendered. Specifically, the foreground image is an image of size (Pw, Ph) (Pw=W) obtained by rendering all elements of the DOM of the web page that were analyzed in step S501. At this time, the image generation server 110 renders the elements selected as foreground with a specific foreground color (for example, solid black) and renders the elements that were not selected as foreground with a specific background color (for example, solid white). In this way, it is possible to obtain a foreground image in which the original layout, including the rendering size of elements other than the foreground, is preserved, and only the pixels rendered by the foreground elements have a specific color.

[0061] In step S503, the image generation server 110 generates a background image. The background image is a page image in which only the elements not selected in step S501 are rendered. Specifically, the background image is an image of size (Pw, Ph) (Pw=W) obtained by rendering all elements of the DOM of the web page that were analyzed in step S501. At this time, the image generation server 110 renders the elements selected as foregrounds in a transparent color and renders the elements not selected as foregrounds in their original colors. In this way, it is possible to obtain a background image in which there are no pixels rendered by foreground elements, while preserving the original layout including the rendering size of the foreground elements.

[0062] In step S504, the image generation server 110 further moves the pixels corresponding to the foreground in the background image generated in step S503 to the corresponding locations in the foreground image generated in step S502. In this embodiment, the targets of the movement are the pixels of characters rendered from image elements, such as parts of illustrations or characters in photographs.

[0063] The character pixel movement process can be performed as follows. First, the image generation server 110 identifies character pixels in the background image. Known character region detection techniques can be used to identify character pixels. For example, the image generation server 110 can extract connected pixel blocks from a target image that has been binarized using Otsu's binarization method, and identify pixels included in pixel blocks that are judged to have a high degree of character-likeness based on size and arrangement as character pixels. Next, for each identified character pixel, the image generation server 110 draws the pixel at the same position on the foreground image with the foreground color. Finally, the image generation server 110 fills in the gap where the character pixel was located on the background image by filling it with the color of the surrounding pixels. Similar to step S501, which pixels to move from the background image to the foreground image is predetermined in the element selection unit 213.

[0064] In step S505, the image generation server 110 compresses the foreground image generated in step S502 and saves the compressed foreground image to the RAM 113 within the image generation server 110. For example, the image generation server 110 compresses and saves the foreground image as a monochrome black binary image in JBIG (Joint Bi-level Image Experts Group) format. The JBIG format is merely an example, and other compression formats such as JBIG2 or MMR (Modified Modified Read) may also be used. In the explanation of step S502, the foreground color is set to a monochrome black, for example, but black and several representative colors may be used as the foreground color, each compressed as a separate binary image, and foreground color information may be added to each binary image. Alternatively, a compression format that allows specifying multiple palette colors, such as PNG (Portable Network Graphics), may be used.

[0065] In step S506, the image generation server 110 compresses the background image generated in step S503 and saves the compressed background image to the RAM 113 within the image generation server 110. For example, the image compression unit 214 compresses and saves the background image in JPEG (Joint Photographic Experts Group) format. The JPEG format is merely an example, and other compression formats such as JPEG2000 may be used.

[0066] Figure 10 shows an example of a page image generated by the image generation server 110 as a result of the processing shown in Figure 5.

[0067] Figure 10(a) shows an example of an original page image that would be generated if the web page to be imaged was converted directly into an image. Figure 10(a) is shown for reference only, and in this embodiment, the image generation server 110 does not actually generate an original page image.

[0068] Figure 10(b) shows an example of a foreground image generated by the image generation server 110 in step S502. In this example, the foreground image has text and button frame elements drawn in a single color.

[0069] Figure 10(c) shows an example of a background image generated by the image generation server 110 in step S503. In this example, the background image contains elements other than text and button frame elements.

[0070] The process shown in Figure 5 and the page image shown in Figure 10 are merely examples, and the image generation server 110 may generate foreground and background images based on other selection methods set in the element selection unit 213. For example, the process in step S504 may be omitted. In this case, the image generation server 110 may skip steps S501 and S502, generate a background image in step S503 by rendering all elements of the DOM tree, and generate a foreground image in step S504 using moved character pixels, etc.

[0071] Returning to the explanation of Figure 3, in step S306, the image generation server 110 transmits the foreground image generated in step S305 to the communication terminal 100. In this example, the compressed image data, which is a monochrome image of full page size (Pw, Ph) compressed in JBIG format and saved in step S505 in Figure 5, is transmitted to the communication terminal 100.

[0072] In step S307, the image generation server 110 expands and extracts the display range (Px, Py) + (W, H) specified in step S301 from the background image generated in step S305, and recompresses the image of the extracted portion. The image generation server 110 then sends the recompressed image to the communication terminal 100. In this example, the area within the target range (Px, Py) + (W, H) is expanded from the image data of the full page size (Pw, Ph) that was compressed and saved in step S506 in Figure 5. The compressed image data obtained by recompressing the expanded size (W, H) image in JPEG format is then sent to the communication terminal 100. At this time, a restart marker may be used to extract and digitize only the image of the target range without recompression.

[0073] If steps S306 and S307 are executed consecutively, the image generation server 110 may respond to the request from the communication terminal 100 by creating a multipart image of the foreground image and the extracted background image.

[0074] After step S307 is completed, the process shown in Figure 3 is finished. The image generation server 110 then waits for the next image generation request from the communication terminal 100, and if an image generation request is received, the process shown in Figure 3 is repeated.

[0075] At this point, for subsequent image generation requests from the communication terminal 100 for the same URL, the image generation server 110 determines in step S302 that page data has already been obtained for the target URL and proceeds to step S307.

[0076] Then, in step S307, the image generation server 110 expands the specified display range from the previously saved background image, recompresses the extracted image, and sends the recompressed image to the communication terminal 100.

[0077] Similarly, the image generation server 110 waits for the next image generation request from the communication terminal 100, and if an image generation request is received, it repeats the process shown in Figure 3.

[0078] Next, the operation of the communication terminal 100 in this embodiment will be described with reference to Figure 4. In the following description of Figure 4, the operation of the image generation server 110 that cooperates with the communication terminal 100 is based on the description with reference to Figures 3 and 5.

[0079] Figure 4 is a flowchart showing an example of processing at a communication terminal according to the first embodiment.

[0080] In step S401, the communication terminal 100 specifies the URL of the web page that the user wants to display by operating the communication terminal 100. The URL is specified, for example, by the user manually entering it using the software keyboard displayed on the display unit 104 of the communication terminal 100. Alternatively, the communication terminal 100 may have a list of URLs stored in advance as bookmarks, and the URL may be specified by the user by touching an item from the bookmark list display. Furthermore, if the communication terminal 100 is already displaying a web page, and the user touches a linked text on the page to specify the destination URL, this is also included in the URL specification in step S401.

[0081] In step S402, the communication terminal 100 requests the image generation server 110 to create an image of the web page specified in step S401. As explained in step S301 in Figure 3, the image generation request to the image generation server 110 includes the URL of the target web page and a display range specification consisting of the display size (W,H) and offset (Px,Py) of the communication terminal 100. In this example, since the request is for an image to be used for the initial display of the target web page, the origin (0,0) is specified as the offset of the display range.

[0082] In step S403, the communication terminal 100 receives the compressed image data of the foreground image sent by the image generation server 110 as a response to the image creation request in step S402, and stores the received compressed image data of the foreground image in the RAM 103 of the communication terminal 100. As explained in step S305 in Figure 3, the received foreground image is compressed image data obtained by compressing a monochrome image of the entire size of the web page (Pw, Ph) (Pw=W, Ph≧H) in JBIG format. An example of a received foreground image is shown in Figure 10(b).

[0083] In step S404, the communication terminal 100 receives the compressed image data of the background image sent by the image generation server 110 in response to the image creation request in step S402, and stores the received compressed image data of the background image in the RAM 103 of the communication terminal 100. In step S404, the same compressed image data of the background image sent from the image generation server 110 in response to the image creation request in step S410, described later, is also received by the communication terminal 100 and stored in the RAM 103 of the communication terminal 100. As explained in step S306 in Figure 3, the received background image is compressed image data obtained by compressing a color image of the display size (W,H) of the communication terminal 100 in JPEG format. An example of a background image received in step S404, which is executed for the first time, is an image cropped from the background image shown in Figure 10(c) above, within the requested display range 1001.

[0084] In step S405, the communication terminal 100 combines the current display range of the foreground image received in step S403 with the background image received in step S404, and displays the combined image on the display unit 104. Specifically, the communication terminal 100 first expands the compressed image data of the background image on RAM 103 into the frame buffer memory (also simply referred to as the frame buffer) used by the display unit 104. An example of the expanded image is shown in Figure 11. The expanded image 1102 corresponds to the image obtained by the image generation server 110 by cutting out the display range 1001 from the image shown in Figure 10(c), which is an example of the background image. Next, the communication terminal 100 expands the image of the display range (0,y)+(W,H) portion from the compressed image data of the foreground image on RAM 103, and overwrites the frame buffer used by the display unit 104 with this image. In the first step, S405, the foreground image 1101 in the initial display range (0,0)+(W,H) is expanded and overwritten. This overwrite is performed by overwriting the corresponding foreground value for pixels that have a foreground pixel value, and doing nothing to pixels that do not have a foreground pixel value. In this example, only pixels with a pixel value of 1 in the monochromatic binary image expanded from compressed image data compressed in JBIG format are overwritten with a predetermined foreground value for pixels at the same position on the background image. For example, the communication terminal 100 overwrites (0,0,0) as the predetermined foreground value (R,G,B). If multiple binary images of different colors are used as foreground images, the communication terminal 100 may overwrite the foreground value assigned to each foreground image. Alternatively, in the case of a palette multicolor image expanded from compressed image data compressed in PNG format, the communication terminal 100 may use only pixels with an opacity of 100% in the alpha channel or palette value as the foreground and overwrite the palette value. As a result of combining the foreground image and background image in step S405, the communication terminal 100 displays an image equivalent to the display range 1011 shown in Figure 10(a).

[0085] From step S406 onward, the communication terminal 100 accepts user input with the page image displayed on the display unit 104. In this example, the communication terminal 100 accepts the following user inputs and performs the corresponding actions. • An action that jumps to the linked URL by tapping on a link within the page. - An action that changes the display range by scrolling up or down, such as by swiping on the screen.

[0086] The communication terminal 100 may perform other actions through operations other than those described above.

[0087] In step S406, the communication terminal 100 determines whether the user operation is a link tap operation. If the user operation is a link tap operation (YES in step S406), the communication terminal 100 completes the page display process for the specified URL and terminates the process shown in Figure 4. Then, the communication terminal 100 specifies the URL of the link destination and starts a new process (the process shown in Figure 4) to display another web page. If the user operation is not a link tap operation (NO in step S406), the communication terminal 100 proceeds to step S407.

[0088] Here, the determination of whether the user operation in S406 is a link tap operation is performed using the cloud browser mechanism. Specifically, the communication terminal 100 sends the page coordinates, obtained by adding the offset of the current display range to the user tap position detected by the input unit 105, to the image generation server 110. The image generation server 110 (its browser unit 211) determines from the acquired page data or image generation result whether there is a link at the specified coordinates, and if there is a link, it responds to the communication terminal 100 with the destination URL of that link. When the communication terminal 100 receives this response, it should determine that the user operation is a link tap operation. This determination method is merely an example, and whether the user operation is a link tap operation may be determined by other methods. For example, the communication terminal 100 may acquire a list of the ranges in the page where links exist and the destination URLs as a response to the image generation request. Then, the communication terminal 100 may use this list to make the determination.

[0089] In step S407, the communication terminal 100 determines whether the user is performing a scroll operation on the display unit 104 (or input unit 105). Scrolling operation refers to, for example, swiping the screen, that is, the state in which the user's finger is touching the touch panel and moving in any direction. If the communication terminal 100 determines that the user is performing a scroll operation (YES in step S407), it proceeds to step S408. If it determines that the user is not performing a scroll operation (NO in step S407), it proceeds to step S409.

[0090] In step S408, the communication terminal 100 updates the page display by changing the display range corresponding to the scroll operation. In this example, the width Pw of the page image generated by the image generation server 110 is equal to the width W of the display unit 104 of the communication terminal 100, so horizontal scrolling is unnecessary. Therefore, the communication terminal 100 interprets the user's vertical scrolling operation as an action to change the display range vertically for a vertically long page. Specifically, when a scroll operation of a movement amount Ys is performed on the display range (0,Y1)+(W,H) before the scroll operation, the communication terminal 100 interprets that the display range has been changed to (0,Y1+Ys)+(W,H). Then, the communication terminal 100 expands the image of the display range (0,Y1+Ys)+(W,H) portion from the compressed image data of the foreground image on RAM 103 and draws the image to the frame buffer used by the display unit 104.

[0091] In this case, the communication terminal 100 does not first load the background image into the frame buffer and then overwrite it with foreground pixels, as in step S405 described above, but instead draws only the foreground image into the frame buffer. Therefore, pixels other than the foreground are displayed on the display unit 104 with a predetermined background color (for example, white). An example of an image involved in the display update performed in conjunction with this scrolling operation is shown as the foreground image 1103 in Figure 11.

[0092] After updating the display, the communication terminal 100 returns to step S406 and repeats the loop processing of steps S406 to S408. The scroll operation detection step S407 in this loop processing may be performed periodically at short intervals, such as 1 / 30th of a second. As a result, the communication terminal 100 can make it appear as if the page display range is changing in synchronization with the user's finger movement. During this loop processing, no communication occurs between the communication terminal 100 and the image generation server 110, so the communication terminal 100 can provide a smooth scrolling display within the limits of its internal image processing performance.

[0093] If it is determined in step S407 that the user is not performing a scroll operation, the communication terminal 100 proceeds to step S409. The first case in which it is determined that the user is not performing a scroll operation is when the user is not touching the touch panel, as in the initial state. Also, if the user lifts their finger from the touch panel at the end of a swipe operation, it is determined that the user is not performing a scroll operation. However, there are cases where it is more appropriate to determine that the scroll operation is continuing (i.e., the user is performing a scroll operation) even after the finger has been lifted, such as when the user immediately resumes a swipe operation in the same direction from a different starting point. For this reason, for example, if a predetermined time such as 1 second has elapsed since the finger was lifted and the user is not performing a scroll operation, the communication terminal 100 may determine that the user is not performing a scroll operation. In addition, if the user swipes at a speed above a certain speed, a so-called flick operation may be realized in which the swipe equivalent motion continues at a reduced speed even after the finger is lifted. If there is inertia in this flick operation, the user is considered to be performing a scroll operation, and when it stops after deceleration, the communication terminal 100 may determine that the user is not performing a scroll operation.

[0094] In step S409, the communication terminal 100 determines whether the current display range has changed from the display range of the page display executed immediately before in step S405. In this determination, the communication terminal 100 may simply compare the two display ranges, or it may determine that the display range has changed if a display update in step S408 has been performed at least once since the execution of step S405. If the current display range has not changed (NO in step S409), the communication terminal 100 resumes the loop processing from step S406. If the current display range has changed (YES in step S409), the communication terminal 100 proceeds to step S410.

[0095] In step S410, the communication terminal 100 sends an image generation request to the image generation server 110, which includes the URL specified in step S401 and the specification of the changed display range (0,Y2)+(W,H), and returns to step S404.

[0096] In the subsequent step S404, the communication terminal 100 receives the background image data for the display range (0,Y2)+(W,H) that the image generation server 110 sent as a response to the image creation request in step S410. The operation of the image generation server 110 at this time corresponds to the case in step S302 in Figure 3 where it determines that the URL has already received page data, proceeds to step S307, extracts the background image for the specified display range, sends it, and then terminates. For example, suppose that in the background image shown in Figure 10(c), the display range 1002 shown in Figure 10(c), which is the bottommost part, is extracted and sent.

[0097] In the subsequent step S405, the communication terminal 100 combines the display range (0,Y2)+(W,H) of the foreground image with the received background image and displays the combined image. As a result, the page image corresponding to the scrolled display range is combined and displayed on the display unit 104. The details of the combination and display process are the same as those in the previous step S405, so the explanation is omitted. Using the example shown in Figure 11, the communication terminal 100 combines the background image 1105 received from the image generation server 110 with the foreground image 1104 expanded from the RAM 103 and displays an image equivalent to the display range 1012 shown in Figure 10(a).

[0098] Furthermore, if the display range in the preceding step S405 matches the display range in step S409, the communication terminal 100 does not need to make the image creation request in step S410. The communication terminal 100 may then execute step S405 using the background image data received in the previous step S404.

[0099] As described above, according to the first embodiment, in response to a request from the communication terminal 100 constituting the cloud browser to generate an image of a web page, the image generation server 110 transmits a foreground image of the entire page and a background image corresponding to the current display range of the communication terminal 100. When no scrolling operation is occurring on the displayed page, the communication terminal 100 combines the foreground image and the background image to provide a page display without any missing information. On the other hand, when a scrolling operation is occurring, the communication terminal 100 displays only the foreground image, enabling display updates that follow the operation. Since no communication occurs during this time, it has the effect of not placing a load on the equipment or communication network. Furthermore, since the foreground image displayed during scrolling contains information on text and button frames, the user can search for the link text to tap and, if not found, continue scrolling. In addition, the compressed image data of the foreground image received by the communication terminal 100 contains only monochrome information such as text and frames, and can be compactly compressed using methods such as JBIG and JBIG2. Therefore, it becomes possible to retain images across the entire page while minimizing RAM consumption and to update the display without receiving images during scrolling.

[0100] <Second Embodiment> In the first embodiment, in step S501 in Figure 5, the image generation server 110 (or its element selection unit 213) selected all the elements in the page data that would draw characters and button frames as foreground elements.

[0101] On the other hand, depending on the content of the web page, if all of those elements are rendered as foreground images, the compressed image data of the foreground images sent from the image generation server 110 may become enormous and may not fit into the RAM 103 of the communication terminal 100.

[0102] Therefore, in the second embodiment, techniques for avoiding or mitigating the above-described situation will be described.

[0103] The following description will focus on the differences from the first embodiment, and will omit explanations of the same or similar configurations and operations as those in the first embodiment.

[0104] In the second embodiment, the element selection unit 213 may set a priority order for the target element types in relation to their impact on the user's browsing operation. The element selection unit 213 may also set an upper limit on the compressed image data size of the foreground image to be transmitted to the communication terminal 100, and select foreground elements based on the priority order and within the range that fits within the upper limit.

[0105] Specifically, the image generation server 110 may actually generate and compress foreground images for each element type, starting from the highest priority, and select the top element types within a range where the sum of the sizes of these compressed image data does not exceed an upper limit. Alternatively, the image generation server 110 may store the relationship between the element type and the compressed data size of the image resulting from its individual rendering, and select the element types using the compressed image data size estimated from the total number of elements of each element type. For example, the communication terminal 100 may maintain a table of relationships between individual character types and the compressed image data size of their individual rendering results for character elements, thereby estimating the compressed image data size for the entire character set on the page.

[0106] The above upper limit may be specified by the communication terminal 100 to the image generation server 110 as the size of the buffer available to the communication terminal 100 for holding the compressed image data of the foreground image on RAM 103. For example, the communication terminal 100 may attach (or include) the upper limit (information) for the compressed image data size to the image creation request from the communication terminal 100 to the image generation server 110. The image generation server 110 may then receive (obtain) the upper limit (information) from the communication terminal 100. Alternatively, by attaching the model ID of the communication terminal 100 to the image creation request, the image generation server 110 may determine the upper limit using a table that associates the model ID with the upper limit of the compressed image data size. Additionally or alternatively, the image generation server 110 may determine the upper limit considering the communication speed between the communication terminal 100 and the image generation server. Limiting the transmission time of the compressed image data of the foreground image has the effect of restricting the waiting time from when the communication terminal 100 requests a web page until the image is displayed.

[0107] Figure 7 shows an example of a table for setting priorities for elements to be included in the foreground.

[0108] In Figure 7, the element with the highest priority is the action-enabled text element. Action-enabled text is, for example, text with a link, which, when selected by the user through an operation such as tapping, triggers a jump to the linked page. Action-enabled text is included in the foreground image with the highest priority so that the communication terminal 100 can jump to the next page immediately after a scroll operation.

[0109] In Figure 7, the second highest priority is non-text action elements. Non-text action elements include, for example, buttons that accept user input and drawing elements that indicate the boundaries of an image. These drawing results serve to assist user interaction by clearly indicating the scope of the action target.

[0110] In Figure 7, the third priority is given to elements such as frames, table borders, and gridlines that do not have any assigned actions. These rendering results show the overall visual layout of the page and provide hints about the page structure, thus assisting the user's browsing experience.

[0111] In Figure 7, the fourth priority is assigned to text elements that have not been given any action. Furthermore, the priority may be subdivided within the fourth-priority text to give higher priority to elements that are emphasized by style-specified size, color, font, etc.

[0112] The priority order shown in Figure 7 is merely an example, and other priority orders may be set. Alternatively, the communication terminal 100 may maintain multiple tables defining different priority orders and select and select the table to use based on the upper size limit specified by the communication terminal 100. Alternatively, the communication terminal 100 may classify the acquired page based on the ratio and arrangement of each type of element it contains, and select and select the table to use based on the classification result of the page.

[0113] As described above, according to the second embodiment, the foreground image displayed by the communication terminal 100, which constitutes the cloud browser, during scrolling contains elements selected according to priority based on their impact on the user's browsing operation. As a result, even when the communication terminal 100 has limited memory, it becomes possible to update the display during scrolling without receiving images and without reducing the user's browsing efficiency.

[0114] <Third Embodiment> In the first embodiment, the image generation server 110 sent compressed image data, which was a compressed foreground image of the entire page, to the communication terminal 100 for the Web page that had been requested to be imaged.

[0115] In contrast, in the third embodiment, if the page size is large, the image generation server 110 may divide the entire page into multiple segments and send compressed image data, obtained by compressing the foreground image for each segment, to the communication terminal 100. As a result, the size of the compressed image data of the foreground image received by the communication terminal 100 is prevented from increasing in proportion to the page size, allowing even devices with small buffer sizes to view large pages.

[0116] The following description will focus on the differences from the first or second embodiment, and will omit explanations of configurations and operations that are the same or similar as those in the first or second embodiment.

[0117] Hereafter, the operation of the image generation server 110 in the third embodiment will be described with reference to Figure 8.

[0118] Figure 8 is a flowchart showing an example of processing by the image generation server according to the third embodiment.

[0119] In step S801, the image generation server 110 receives a request from the communication terminal 100 to create an image of a web page. In this example, the image creation request received in step S801 includes the URL of the target web page and a specification of the display range of the page image, as well as a specification of whether the target of the request is the foreground, background, or both.

[0120] In step S802, the image generation server 110 determines whether or not page data has already been retrieved for the requested target web page URL. If the page data has already been retrieved (YES in step S802), the image generation server 110 proceeds to step S805; if the page data has not already been retrieved (NO in step S802), the image generation server 110 proceeds to step S803.

[0121] In step S803, the image generation server 110 obtains page data from the URL of the web page.

[0122] In step S804, the image generation server 110 records the current time as the time the page data for the target URL is acquired, associating it with the target URL.

[0123] In step S805, the image generation server 110 determines whether the image generation request received in step S801 targets a foreground image. If the image generation request targets a foreground image (YES in step S805), the image generation server 110 proceeds to step S806; otherwise, it proceeds to step S809.

[0124] In step S806, the image generation server 110 determines which segment of the page image the requested display range corresponds to. Here, a segment is each fragment when the page image is divided from the top in a predetermined range where the height direction is below the upper limit value Sh with respect to the size (Pw, Ph) of the image obtained by imaging the entire web page.

[0125] Segment division has the effect of suppressing the size of the compressed image data received by the communication terminal 100 by creating compressed image data of the foreground image for each segment. The segment height Sh may be, for example, a value obtained by multiplying the page width Pw by a predetermined value such as 8 times. Alternatively, in the case of a page with a large amount of information per area where the size of the compressed image data is predicted to be large, a smaller Sh may be set to increase the number of segments and reduce the amount of data per segment.

[0126] For example, when Ph = Sh×N + m (N is an integer such that m < Sh), segment 0 has the range (0, 0)+(Pw, Sh), segment 1 has the range (0, Sh)+(Pw, Sh), and the last segment N has the range (0, Sh×N)+(Pw, m). In this case, which segment the requested display range corresponds to may be determined by which segment range the upper left coordinate of the display range is included in. This determination method is merely an example. For example, which segment the requested display range corresponds to may be determined using the lower right or center coordinates instead of the upper left coordinates.

[0127] In step S807, the image generation server 110 generates a foreground image of the segment range determined in step S806. The foreground image is generated in the same manner as the processes in steps S501 to S502 in Figure 5 described above, and is compressed and saved in the same manner as in step S505. Alternatively, when generating a foreground image of the segment range, the image generation server 110 may first image the entire page and then crop the relevant range, similar to the process shown in Figure 5. Or, the element selection unit 213 may select elements to be drawn within the segment range from the page data, and the browser unit 211 may only image the relevant portion.

[0128] In step S808, the image generation server 110 transmits the compressed image data of the foreground image of the segment generated in step S807 to the communication terminal 100. At this time, segment configuration information may also be attached, indicating how many segments exist for the page for which image generation was requested and which segment the transmitted segment is.

[0129] In step S809, the image generation server 110 determines whether the image generation request received in step S801 targets a background image. If the image generation request targets a background image (YES in step S809), the image generation server 110 proceeds to step S810. If the image generation request does not target a background image (NO in step S809), the server terminates the process shown in Figure 8 and waits for the next image generation request.

[0130] In step S810, the image generation server 110 generates a background image of the display area specified in the image creation request in step S801. The background image is generated in the same manner as the processes in steps S501 and S503 in Figure 5 described above, and is compressed and saved in the same manner as in step S506. When generating the background image of the display area, the image generation server 110 may first image the entire segment and then cut out the relevant area. Alternatively, the element selection unit 213 may select elements to be drawn within the display area from the page data, and the browser unit 211 may only image the relevant portion.

[0131] In step S811, the image generation server 110 sends the compressed image data of the background image generated in step S810 to the communication terminal 100 for the display range of the image generation request. After that, the image generation server 110 completes the process shown in Figure 8 and waits for the next image generation request.

[0132] Next, the operation of the communication terminal 100 in the third embodiment will be described with reference to Figure 9. In the following description of Figure 9, the operation of the image generation server 110 that cooperates with the communication terminal 100 will be based on the description with reference to Figure 8.

[0133] Figure 9 is a flowchart showing an example of processing at a communication terminal according to the third embodiment.

[0134] In step S901, the communication terminal 100 specifies the URL of the web page that the user wants to display by operating the communication terminal 100. The process in step S901 is the same as the process in step S401 in Figure 4.

[0135] In step S902, the communication terminal 100 requests the image generation server 110 to generate an image of the web page specified in step S901. The image generation request to the image generation server 110 includes the URL of the target web page and the display range specification, as well as a specification of whether to request the foreground, background, or both of the above to be imaged. In this example, since the image to be used for the initial display of the target page is requested in step S902, the display range is specified as the range from the origin (0,0) to the display size (W,H). In addition, both the foreground and background are specified as the images to be imaged.

[0136] In step S903, the communication terminal 100 receives the compressed image data of the foreground image generated and transmitted by the image generation server 110 as a response to the image generation request in step S902, and stores the received compressed image data in RAM 103. The display range specified in step S902 is (0,0)+(W,H), and the image generation server 110, upon receiving the request, interprets it as an image generation request for segment 0 with respect to the origin of the display range (0,0). The communication terminal 100 then receives the compressed image data of the foreground image in the range (0,0)+(Pw,Sh) of segment 0 that was generated and transmitted. At this time, the communication terminal 100 also receives segment configuration information from the image generation server 110, indicating how many segments exist on the page and which segment the received segment is. In this example, the communication terminal 100 receives segment configuration information indicating that there are N+1 segments from segment 0 to segment N, and the received foreground image is segment 0, which is segment 0.

[0137] In step S904, the communication terminal 100 receives the compressed image data of the background image generated and transmitted by the image generation server 110 in response to the image generation request in step S902, and stores the received compressed image data in RAM 103. In this example, the communication terminal 100 receives the compressed image data of the background image for the initial display range (0,0)+(W,H). In step S904, the same compressed image data of the background image transmitted from the image generation server 110 in response to the image generation request in step S910 (described later) is also received by the communication terminal 100 and stored in RAM 103 of the communication terminal 100.

[0138] In step S905, the communication terminal 100 combines the current display range of the foreground image received in step S403 with the background image received in step S904, and displays the combined image on the display unit 104. The processing details for the combination in step S905 are the same as those in step S405 in Figure 4, so the explanation is omitted.

[0139] In step S906, the communication terminal 100 determines whether the user operation is a link tap operation, similar to step S406 in Figure 4. If the user operation is a link tap operation (YES in step S906), the communication terminal 100 completes the page display process for the specified URL and terminates the process shown in Figure 9. Then, the communication terminal 100 specifies the URL of the link destination and starts a new process (the process shown in Figure 9) to display another web page. If the user operation is not a link tap operation (NO in step S906), the communication terminal 100 proceeds to step S907.

[0140] In step S907, the communication terminal 100 determines whether the user is currently performing a scroll operation on the display unit 104 (or input unit 105), similar to step S407 in Figure 4. If the communication terminal 100 determines that the user is performing a scroll operation (YES in step S907), it proceeds to step S911. If it determines that the user is not performing a scroll operation (NO in step S907), it proceeds to step S909.

[0141] In step S911, the communication terminal 100 determines whether the display range of the scroll destination is outside the segment of the received foreground image. The display range of the scroll destination is determined to be outside the segment in the following cases, for example: 1. When scrolling downwards, if the page coordinates corresponding to the bottom right of the currently displayed area are outside the segment range. 2. When scrolling upwards, if the page coordinates corresponding to the top left of the currently displayed area are outside the segment range.

[0142] If the communication terminal 100 determines that the display range of the scroll destination is outside the segment of the received foreground image (YES in step S911), it proceeds to step S912. On the other hand, if the communication terminal 100 determines that the display range of the scroll destination is not outside the segment of the received foreground image (NO in step S911), it proceeds to step S908.

[0143] In step S912, the communication terminal 100 uses the segment configuration information obtained when the foreground image was received in step S903 to determine whether or not there is a next segment to be imaged as the scroll destination. In step S912, it is also determined whether or not there is a next segment to be imaged as the scroll destination to be imaged, using the segment configuration information obtained when the foreground image was received in step S914, which will be described later. For example, if segments 0 to N exist and the current segment is the Kth segment in a downward scroll operation, the communication terminal 100 determines that a next segment exists if K+1 ≤ N. Alternatively, the communication terminal 100 determines that a next segment exists if the current segment is not the 0th segment in an upward scroll operation. If a next segment exists (YES in step S912), the communication terminal 100 proceeds to step S913. If a next segment does not exist (NO in step S912), it cannot scroll any further, so it returns to step S906 without updating the display.

[0144] In step S913, the communication terminal 100 sends an image generation request to the image generation server 110 to obtain an image of the foreground of the next segment, including the scroll destination. As explained in steps S806 to S808 in Figure 8, in this embodiment, the image generation server 110, in response to an image generation request targeting the foreground, generates an image of the foreground of the segment including the upper-left coordinate of the specified display range and transmits the compressed image data. Here, if the current segment is the Kth segment and scrolling is downward, the communication terminal 100 can request an image of the foreground of the next segment by specifying the range (0, Sh × (K + 1)) + (W, H) using the acquired segment configuration information. If the current segment is the Kth segment and scrolling is upward, the communication terminal 100 can request an image of the foreground of the next segment by specifying the range (0, Sh × (K - 1)) + (W, H) using the acquired segment configuration information.

[0145] In step S914, the communication terminal 100 receives the foreground image of the next segment generated and transmitted by the image generation server 110 as a response to the image generation request in step S913. At this time, the communication terminal 100 obtains segment configuration information, including which segment the received segment is, as in step S903.

[0146] In step S908, the communication terminal 100 changes the display range corresponding to the scroll operation and updates the page display. If the process proceeds from step S914 to step S908, a portion of the display range at the scroll destination will be included in the range of the next segment received in step S914. In that case, the communication terminal 100 can extract and stitch together the foreground images of the corresponding ranges from the received segment and the newly received segment, and then display the stitched image on the display unit 104.

[0147] If it is determined in step S907 that the user is not currently scrolling, then in step S909, the communication terminal 100 performs the same processing as in step S409 in Figure 4. That is, the communication terminal 100 determines whether the current display range has changed from the display range of the page display performed immediately before in step 905. If the current display range has not changed (NO in step S909), the communication terminal 100 resumes the loop processing from step S906. If the current display range has changed (YES in step S909), the communication terminal 100 proceeds to step S910.

[0148] In step S910, the communication terminal 100 sends an image generation request to the image generation server 110, which includes the URL specified in step S401 and a specification of the display range to scroll to. In this case, only the background is specified as the target of the image generation request. Then, the communication terminal 100 returns to step S904.

[0149] In the subsequent step S904, the communication terminal 100 receives the background image data sent by the image generation server 110 as a response to the image creation request in step S910.

[0150] In the following step S905, the communication terminal 100 combines an image obtained by cutting out the display range of the scroll destination from the foreground image received in step S903 or S914 with the background image received in the preceding step S904, and displays the combined image.

[0151] As described above, according to the third embodiment, the compressed image data of the foreground image received when viewing a large page is limited to data within the range obtained by dividing the entire page into multiple segments. As a result, the buffer size for receiving the foreground image data of the communication terminal 100 can be reduced.

[0152] Furthermore, in the third embodiment, it is possible to update the display without receiving an image during a scroll operation within a received segment. Therefore, it has the effect of reducing the memory usage of the communication terminal 100 while not reducing the browsing efficiency that involves scrolling operations.

[0153] <Fourth Embodiment> In the first embodiment, in step S305 in Figure 3, the image generation server 110 generated a foreground image in which the foreground elements selected by the element selection unit 213 were drawn, and a background image in which elements other than the foreground were drawn. Then, in step S405 in Figure 4, the communication terminal 100 displayed a composite image on the display unit 104 in which pixels having foreground values ​​in the foreground image were overwritten with the background image.

[0154] In contrast, in the fourth embodiment, the image generation server 110 performs the processing shown in Figure 3, but in step S305, it may generate the same foreground image as in the first embodiment, as well as a background image in which all elements, including the foreground elements, are drawn.

[0155] The following description will focus on the differences from the first to third embodiments, and will omit explanations of configurations and operations that are the same or similar as those of the first to third embodiments.

[0156] The operation of the image generation server 110 according to the fourth embodiment, including the generation of a background image, will be described below with reference to Figure 12.

[0157] Figure 12 is a flowchart showing a detailed example of step S305 in Figure 3 according to the fourth embodiment.

[0158] In step S1201, the image generation server 110 analyzes the page data acquired in step S303 in Figure 3 and selects foreground elements to be rendered. The processing in step S1201 is the same as in step S501 in Figure 5, so the explanation is omitted.

[0159] In step S1202, the image generation server 110 generates a foreground image in which only the selected foreground elements are drawn. The processing in step S1202 is the same as in step S502 in Figure 5, so the explanation is omitted.

[0160] In step S1203, the image generation server 110 generates a background image in which all elements of the page data are rendered.

[0161] In step S1205, the image generation server 110 compresses the foreground image generated in step S1202 and saves the compressed foreground image to the RAM 113 within the image generation server 110. The compression method in step S1205 is the same as in step S505 in Figure 5.

[0162] In step S1206, the image generation server 110 compresses the background image generated in step S1203 and stores the compressed background image in RAM 113 within the image generation server 110. The compression method in step S1206 is the same as the method used for the background image in step S506 in Figure 5.

[0163] In the fourth embodiment, in step S307 in Figure 3, the image generation server 110 extracts the display range from the compressed image data of the foreground background image saved in step S1206, recompresses it, and transmits the recompressed image to the communication terminal 100. At this time, the image generation server 110 adds a flag indicating that what is being transmitted is a foreground background image and not a background image, and then transmits the image.

[0164] In the fourth embodiment, the communication terminal 100 performs the process shown in Figure 4, but in step S405, it performs the process shown in Figure 13. The operation of the communication terminal 100 according to the fourth embodiment will be described in detail below with reference to Figure 13.

[0165] Figure 13 is a flowchart showing a detailed example of step S405 in Figure 4, according to the fourth embodiment.

[0166] In step S1301, the communication terminal 100 determines whether the image received from the image generation server 110 as a background image has a flag (called the background flag) indicating that it is a previous background image. If the background flag is present (YES in step S1301), the communication terminal 100 proceeds to step S1302; if the background flag is not present (NO in step S1301), it proceeds to step S1303.

[0167] In step S1302, the communication terminal 100 assumes that the image data received from the image generation server 110 is compressed image data of the foreground image, and displays the unfolded image on the display unit 104. At this time, the communication terminal 100 does not perform any synthesis with the previously received foreground image.

[0168] On the other hand, in step S1303, the communication terminal 100 displays the composite image on the display unit 104 in the manner described in step S405 in Figure 4 above. That is, the communication terminal 100 extracts the display range portion of the foreground image, combines the extracted display range portion of the foreground image with the background image received in step S1301, and displays the resulting composite image on the display unit 104.

[0169] As described above, according to the fourth embodiment, in response to a request from the communication terminal 100 to generate an image of a web page, the image generation server 110 transmits to the communication terminal 100 a foreground image containing information that the user is interacting with, and a background image containing both the foreground and background. At this time, the foreground image is initially transmitted once as compressed image data of the entire page, and the background image is transmitted each time the communication terminal 100 updates its display range as compressed image data of the images within that range. When no scrolling operation is occurring on the displayed page, the communication terminal 100 displays the background image, providing a page display with no missing information. On the other hand, when a scrolling operation is occurring, the communication terminal 100 displays only the foreground image, which has the effect of not reducing browsing efficiency by updating the display in accordance with the operation. In addition, since no communication occurs during this time, it has the effect of not placing a load on the equipment or communication network.

[0170] <Fifth Embodiment> In the fifth embodiment, the image generation server 110 may select (or decide) whether to generate a background image that does not include a foreground, as in the first embodiment, or a foreground background image that includes a foreground, as in the fourth embodiment, depending on the actual page content. The image generation server 110 may then reflect the selection result in the foreground background flag described above and send the foreground background flag along with the image to the communication terminal 100. The foreground background flag can be described as information indicating whether or not the image includes the rendering result of a foreground element. A value of 1 for the foreground background flag means that the image includes the rendering result of a foreground element, and a value of 0 for the foreground background flag means that the image does not include the rendering result of a foreground element. The values ​​1 and 0 may have opposite meanings. For example, the browser unit 211 may decide whether or not to include the rendering result of a foreground element in the image based on the page content and reflect the decision in the foreground background flag.

[0171] For example, if the text on a page has various decorations such as shadows or indentations, a single-color or multi-color foreground image, as described in step S504 in Figure 5 above, may not be able to reproduce the original text. Alternatively, when using the symbol mode of JBIG2 to obtain a high compression ratio, the differences between fonts may not be reproduced in the unfolded foreground image on pages that use multiple similar fonts. For pages where this is possible, the image generation server 110 may generate a foreground background image that includes the foreground, and for other pages, it may generate a background image that does not include the foreground.

[0172] The image generation server 110 may further divide the page into segments, as in the second embodiment, and generate either a foreground background image or a background image for each segment. In this case, as described in step S1301 in Figure 13 in the fourth embodiment, the communication terminal 100 decides, based on the foreground background flag, whether to display the foreground background image as is or to display a composite image which is a combination of the foreground image and the background image.

[0173] As described above, according to the fifth embodiment, when no scrolling operation is performed on the displayed page, the communication terminal 100 provides a page display that includes both foreground and background information. At this time, the communication terminal 100 provides either a foreground background image or a composite image of the foreground and background images, according to the content of the image generated by the image generation server 110. On the other hand, when no scrolling operation is performed, the communication terminal 100 displays only the foreground image, which has the effect of not reducing browsing efficiency by updating the display in response to the operation. In addition, since no communication occurs during this time, it has the effect of not placing a load on the equipment or communication network.

[0174] <Other Embodiments> In the first embodiment, the image generation server 110 sent compressed image data, which was a compressed foreground image of the entire page, to the communication terminal 100 for the Web page that had been requested to be imaged. On the other hand, in the third embodiment, the image generation server 110 divided the entire page into multiple segments and sent compressed image data, which was a compressed foreground image of each segment, to the communication terminal 100.

[0175] In response to this, the image generation server 110 may choose to transmit compressed image data in which the foreground image of the entire page is compressed, as in the first embodiment, or to transmit compressed image data in which the foreground image of each segment is compressed, as in the third embodiment. The image generation server 110 may choose which compressed image data to transmit based, for example, on the relationship between the page width Pw and the page height Ph, or on the amount of information per unit area on the page. For example, the image generation server 110 may transmit compressed image data as in the first embodiment if Ph is less than or equal to Pw*N (for example, N is a predetermined integer of 2 or more), and transmit compressed image data as in the third embodiment otherwise. Also, for example, the image generation server 110 may transmit compressed image data as in the first embodiment if the amount of information per unit area on the page is less than or equal to a predetermined amount, and transmit compressed image data as in the third embodiment otherwise. For example, the browser unit 211 may choose which compressed image data to transmit. In other words, the browser unit 211 may decide whether to set the target area of ​​the compressed image data to the entire web page, or to set it to each of the two or more regions (segments) into which the entire web page is divided.

[0176] The same effects as those of the embodiments described above can be obtained with the above configuration as well.

[0177] Alternatively, a storage medium containing program code for software that implements the above-described functions may be supplied to a system or device, and the computer (CPU, MPU) of the system or device may read and execute the program code stored on the storage medium. In this case, the program code read from the storage medium itself will implement the functions of the above-described embodiment, and the storage medium containing that program code will constitute the above-described device.

[0178] For supplying program code, storage media such as flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, DVDs, etc., can be used.

[0179] Furthermore, the above-mentioned functions may be realized not only by the computer executing the program code it reads, but also by the operating system running on the computer performing some or all of the actual processing based on the instructions of that program code.

[0180] Furthermore, the program code read from the storage medium is written to the memory of a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of that program code, the CPU of the function expansion board or function expansion unit may perform some or all of the actual processing to realize the above-mentioned functions.

[0181] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, this disclosure can also be implemented by a circuit (for example, an application-specific integrated circuit (ASIC)) that implements one or more functions.

[0182] Furthermore, some of the processes described in this disclosure with reference to the flowchart may be implemented in hardware. For example, by using a predetermined compiler, dedicated circuits can be automatically generated on a field-programmable gate array (FPGA) from the program to implement each step. Alternatively, a gate array circuit may be formed in the same way as an FPGA and implemented in hardware.

[0183] The names of the functional units, messages, parameters, fields, etc., described in the embodiments described above may be changed to other names.

[0184] The order of the processing procedures, sequences, flowcharts, etc., in the embodiments described above is not limited to the specific order presented, and may be rearranged or additional steps may be added, as long as they do not contradict each other.

[0185] Matters described in one embodiment may be incorporated into another embodiment, insofar as they do not contradict each other.

[0186] Furthermore, the following additional information is disclosed regarding the above embodiments.

[0187] [Note 1] An information processing system having an information processing device and a terminal device, The aforementioned information processing device is A selection method for selecting the primary element contained in a web page, A generation means for generating a first image which is the result of drawing only the first element within a first area of ​​the web page, and a second image which includes the result of drawing a second element other than the first element within a second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device, A transmission means for transmitting the first image and the second image to the terminal device, It has, The aforementioned terminal device is Receiving means for receiving the first image and the second image from the information processing device, A display control means that, based on the occurrence of a scroll operation on the terminal device, extracts a third image from the second region of the first image and displays the third image, It has, An information processing system in which, based on the fact that no scrolling operation has occurred on the terminal device, the receiving means receives the second image corresponding to the change in the display range from the information processing device, and the display control means causes the second image corresponding to the change in the display range to be displayed.

[0188] [Note 2] The generation means generates a second image which includes the drawing result of the second element but does not include the drawing result of the first element. The information processing system according to Appendix 1, wherein, based on the fact that no scrolling operation has occurred on the terminal device, the display control means causes the second image and the third image to be displayed in combination according to the change in the display range.

[0189] [Note 3] The system further comprises a first determination means for determining whether or not to include the drawing result of the second element in the second image. Based on the fact that no scrolling operation has occurred on the terminal device, the display control means displays the second image in accordance with the change in the display range if the second image includes the drawing result of the second element, and displays a composite image of the second image and the third image in accordance with the change in the display range if the second image does not include the drawing result of the second element, as described in Appendix 1.

[0190] [Note 4] The transmitting means transmits to the terminal device information indicating whether or not the second image includes the drawing result of the second element. The receiving means is an information processing system as described in Appendix 3, which receives the information from the information processing device.

[0191] [Note 5] The first element is an information processing system as described in any of the appendices 1 to 4, which includes one or more of the following: characters, table frames, input frames, and button frames.

[0192] [Note 6] The selection means is an information processing system according to any one of the appendices 1 to 5, which selects the first element based on a priority order set for the element type.

[0193] [Note 7] The information processing system according to any one of the appendices 1 to 6, wherein the selection means selects the first element based on an upper limit for the data size of the first image.

[0194] [Note 8] The information processing system according to Appendix 7, further comprising means for obtaining the aforementioned upper limit from the terminal device.

[0195] [Note 9] The sorting means is an information processing system according to any one of the appendices 1 to 8, which sorts the first element based on the classification result of the web page.

[0196] [Note 10] The first domain is the entire web page, as described in any of the appendices 1 to 9 of the information processing system.

[0197] [Note 11] The first region is each of the regions obtained by dividing the entire web page into two or more regions, as described in any of the appendices 1 to 9 of the information processing system.

[0198] [Note 12] The information processing system according to any one of the appendices 1 to 9, further comprising a second determination means for determining whether the first area is set to the entire area of ​​the web page or to each of the areas obtained by dividing the entire area of ​​the web page into two or more parts.

[0199] [Note 13] A selection method for selecting the primary element contained in a web page, A generation means for generating a first image which is the result of drawing only the first element within a first area of ​​the web page, and a second image which includes the result of drawing a second element other than the first element within a second area which is narrower than the first area of ​​the web page and corresponds to the display range of a terminal device, A transmission means for transmitting the first image and the second image to the terminal device, An information processing device having

[0200] [Note 14] A terminal device, Receiving means for receiving from an information processing device a first image which is the result of rendering only the first element within a first area of ​​a web page, and a second image which includes the result of rendering the second element other than the first element within a second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device, A display control means that, based on the occurrence of a scroll operation on the terminal device, extracts a third image from the second region of the first image and displays the third image, It has, A terminal device wherein, based on the fact that no scrolling operation has occurred on the terminal device, the receiving means receives the second image corresponding to the change in the display range from the information processing device, and the display control means causes the terminal device to display the second image corresponding to the change in the display range.

[0201] [Note 15] A method for controlling an information processing device, The process of selecting the primary element to be included in a web page, A step of generating a first image which is the result of drawing only the first element within the first area of ​​the web page, and a second image which includes the result of drawing a second element other than the first element within a second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device, A step of transmitting the first image and the second image to the terminal device, A control method including

[0202] [Note 16] A method for controlling a terminal device, The process of receiving from an information processing device a first image which is the result of rendering only the first element within a first area of ​​a web page, and a second image which includes the result of rendering the second element other than the first element within a second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device, Based on the fact that a scroll operation is occurring on the terminal device, the process involves extracting a third image from the second region of the first image and displaying the third image. Based on the fact that no scroll operation has occurred on the terminal device, A step of receiving the second image corresponding to the change in the display range from the information processing device, A step of displaying the second image corresponding to the change in the display range, A control method including

[0203] [Note 17] A program that causes a computer to perform the control method described in Appendix 15 or 16. [Explanation of Symbols]

[0204] 100 communication terminals 110 Image generation server

Claims

1. An information processing system having an information processing device and a terminal device, The aforementioned information processing device is A selection method for selecting the primary element included in a web page, A generation means for generating a first image which is the result of drawing only the first element within the first region of the web page, and a second image which includes the result of drawing a second element other than the first element within the second region which is narrower than the first region of the web page and corresponds to the display range of the terminal device, A transmission means for transmitting the first image and the second image to the terminal device, It has, The aforementioned terminal device is Receiving means for receiving the first image and the second image from the information processing device, A display control means that, based on the occurrence of a scroll operation on the terminal device, extracts a third image from the second region of the first image and displays the third image, It has, An information processing system in which, based on the fact that no scrolling operation has occurred on the terminal device, the receiving means receives the second image corresponding to the change in the display range from the information processing device, and the display control means causes the second image corresponding to the change in the display range to be displayed.

2. The generation means generates a second image which includes the drawing result of the second element but does not include the drawing result of the first element. The information processing system according to claim 1, wherein, based on the fact that no scrolling operation has occurred on the terminal device, the display control means causes the second image and the third image to be displayed in combination according to the change in the display range.

3. The system further comprises a first determination means for determining whether or not to include the drawing result of the second element in the second image. The information processing system according to claim 1, wherein, based on the fact that no scrolling operation has occurred on the terminal device, the display control means displays the second image in accordance with the change in the display range if the second image includes the drawing result of the second element, and displays a composite image of the second image and the third image in accordance with the change in the display range if the second image does not include the drawing result of the second element.

4. The transmitting means transmits to the terminal device information indicating whether or not the second image includes the drawing result of the second element. The information processing system according to claim 3, wherein the receiving means receives the information from the information processing device.

5. The information processing system according to claim 1, wherein the first element includes one or more of characters, table frames, input frames, and button frames.

6. The information processing system according to claim 1, wherein the selection means selects the first element based on a priority order set for the element species.

7. The information processing system according to claim 1, wherein the selection means selects the first element based on an upper limit value for the data size of the first image.

8. The information processing system according to claim 7, further comprising means for obtaining the upper limit from the terminal device.

9. The information processing system according to claim 1, wherein the selection means selects the first element based on the classification result of the web page.

10. The information processing system according to claim 1, wherein the first area is the entire area of ​​the web page.

11. The information processing system according to claim 1, wherein the first region is each of the regions obtained by dividing the entire web page into two or more regions.

12. The information processing system according to claim 1, further comprising a second determination means for determining whether to set the first area to the entire area of ​​the web page or to set it to each of the areas obtained by dividing the entire area of ​​the web page into two or more areas.

13. A selection method for selecting the primary element included in a web page, A generation means for generating a first image which is the result of drawing only the first element within the first area of ​​the web page, and a second image which includes the result of drawing a second element other than the first element within a second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device, A transmission means for transmitting the first image and the second image to the terminal device, An information processing device having

14. A terminal device, Receiving means for receiving from an information processing device a first image which is the result of drawing only the first element within the first area of ​​a web page, and a second image which includes the result of drawing the second element other than the first element within the second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device, A display control means that, based on the occurrence of a scroll operation on the terminal device, extracts a third image from the second region of the first image and displays the third image, It has, A terminal device wherein, based on the fact that no scrolling operation has occurred on the terminal device, the receiving means receives the second image corresponding to the change in the display range from the information processing device, and the display control means causes the terminal device to display the second image corresponding to the change in the display range.

15. A method for controlling an information processing device, The process of selecting the primary element to be included in the web page, A step of generating a first image which is the result of drawing only the first element within the first area of ​​the web page, and a second image which includes the result of drawing a second element other than the first element within the second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device, A step of transmitting the first image and the second image to the terminal device, A control method including

16. A method for controlling a terminal device, The process of receiving from an information processing device a first image which is the result of drawing only the first element within the first area of ​​a web page, and a second image which includes the result of drawing the second element other than the first element within the second area which is narrower than the first area of ​​the web page and corresponds to the display range of the terminal device, Based on the fact that a scroll operation is occurring on the terminal device, the process involves extracting a third image from the second region of the first image and displaying the third image. Based on the fact that no scroll operation has occurred on the terminal device, A step of receiving the second image corresponding to the change in the display range from the information processing device, A step of displaying the second image corresponding to the change in the display range, A control method including

17. A program for causing a computer to execute the control method described in claim 15 or 16.

Images