Information processing system, information processing device, method for controlling information processing device, and program

Abstract

To provide a technique capable of shortening a processing time and early recognizing an element selected by a user.SOLUTION: An information processing device for receiving, from another information processing device, drawing results obtained by processing a Web content by the other information processing device and displaying the drawing results, comprises: transmission means for transmitting information related to a user operation performed on the information processing apparatus to the other information processing device; reception means for receiving a response from the other information processing device; and drawing means for drawing a user interface related to the user operation on the Web content on the basis of detailed information when the response includes the detailed information of an element which is a target of the user operation.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, a control method and a program for an information processing device, and in particular to a technique related to UI display control. [Background technology]

[0002] Traditionally, users have used web browsers to browse web pages. Web browsers are now installed not only on personal computers but also on mobile devices such as smartphones and tablets, making it common to browse web pages while on the go.

[0003] A web browser retrieves web content specified by a URL (HTML documents, JavaScript, CSS, etc.), analyzes the web content, generates rendering results through rendering processing, and displays the web content specified by the user. In addition, when a user operates the rendering results via a mouse, touch panel, keyboard, etc., the browser displays the rendering results in response to that operation. The program that takes web content as input and processes the rendering results of the web content is referred to here as the browser engine. The browser engine, the processing of user operations, and the output processing of the rendering results generated by the browser engine are generally all performed within a single device; hereafter, this type of web browser will be referred to as an embedded browser.

[0004] On the other hand, resource-saving devices may not be able to be equipped with an embedded browser. In contrast to this, there is a cloud browser format in which rendering results are generated on a server with a browser engine and displayed on a client device. The client device mainly handles input and output, such as specifying a URL, displaying rendering results, and user operations. The cloud server acquires web content, analyzes the web content, and generates rendering results through rendering processing. In this way, distributing processing allows web content to be displayed and operated even on resource-saving devices. For example, rendering results are converted into images on the server, and the client device displays the images sent from the server.

[0005] Patent Document 1 discloses that the rendering results generated on the server side are visualized and displayed on a client device. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-182793 Summary of the Invention [Problem to be solved by the invention]

[0007] Incidentally, as a rendering result, a momentary change in UI is displayed in response to a user operation, and then the final result of a transition to another URL is displayed. For example, in HTML, the a element can be used to specify a link destination URL. When this element is clicked, a typical web browser will momentarily display the color of the clicked element and a rectangle surrounding the element to indicate which element was clicked. After that, the link destination of the a element is displayed as the final rendering result.

[0008] However, when rendering to display a momentary UI change in a cloud browser, it is necessary to generate the rendering results for that momentary UI change, convert them into a video or image, and send them to the client device. This means that it takes time to display the final transition result. On the other hand, if the display of momentary UI changes is omitted, the user will not be able to recognize where they clicked (or touched). As a result, if they select (press) the wrong element, they will not notice until after the rendering is complete.

[0009] The present invention has been made in view of the above-mentioned problems, and aims to provide a technique that enables an element selected by a user to be recognized early while shortening processing time. [Means for solving the problem]

[0010] An information processing system according to the present invention that achieves the above object comprises: An information processing system in which a first information processing device processes web content and transmits a rendering result obtained to a second information processing device for display, The first information processing device a receiving means for receiving information regarding a user operation on the web content from the second information processing device; an element acquisition means for acquiring an element that is a target of the user operation; an information acquisition means for acquiring detailed information of the element when the element is a predetermined element; a transmitting means for transmitting the detailed information to the second information processing device, The second information processing device receiving means for receiving the detailed information; a drawing means for drawing a user interface relating to the user operation on the web content based on the detailed information; The present invention is characterized by comprising: [Effects of the Invention]

[0011] According to the present invention, it is possible to reduce processing time and quickly recognize an element selected by a user. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the hardware configuration of a cloud server according to an embodiment. [Figure 3] FIG. 2 is a block diagram illustrating the hardware configuration of a client device according to an embodiment. [Figure 4] FIG. 2 is a functional block diagram of a client device according to one embodiment. [Figure 5] FIG. 2 is a functional block diagram of a cloud server according to an embodiment. [Figure 6] 10 is a flowchart showing the flow of processing by a cloud server according to an embodiment. [Figure 7] 10 is a flowchart illustrating a processing flow of a client device according to an embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of UI rendering on a client device according to an embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of UI rendering on a client device according to an embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of UI rendering on a client device according to an embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of a predetermined element according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] (Embodiment 1) In this embodiment, we describe a cloud browser that generates and displays some UI (user interface) changes on the client device by sending information about a selected (clicked or touched) element from a cloud server to the client device.

[0015] <System configuration> 1 is a diagram showing the configuration of an information processing system according to this embodiment, and is a diagram showing the configuration of a cloud browser. Reference numeral 100 denotes an information processing system including a cloud server 101 and a client device 102.

[0016] Reference numeral 101 denotes an information processing device that functions as a cloud server. The cloud server 101 includes a browser engine 103 and an imaging program 104. The cloud server 101 runs on a virtual machine in a cloud environment. Various device resources required for a PC are virtually allocated to the virtual machine. Therefore, the browser engine running on the virtual machine can operate in the same way as when running on a PC.

[0017] An information processing device 102 functions as a client device on the user side who browses a web page. The client device 102 transmits user operations to the cloud server 101, and displays images of web pages generated by the cloud server 101.

[0018] Reference numeral 103 denotes a browser engine. The browser engine 103 analyzes web content obtained from a web content server 105, performs drawing processing, etc., and creates the drawing results of a web page. The browser engine 103 analyzes the syntax of an HTML document and CSS, and generates a DOM tree and a CSSOM (CSS Object Model) that represent each element as a tree structure. The browser engine 103 further generates a render tree that represents the drawing structure based on these trees. The browser engine 103 also analyzes and compiles JavaScript to convert it into machine language and executes the script.

[0019] An imaging program 104 converts the rendering results generated by the browser engine 103 into image data in an image format such as PNG or JPEG. A web content server 105 provides web content such as HTML, JavaScript, and CSS based on requests.

[0020] <Cloud server hardware configuration> 2 is a block diagram showing the hardware configuration of the cloud server 101 according to this embodiment. The cloud server 101 includes, as its main components, a controller unit 201 that controls the entire cloud server 101, a storage device 205, and a communication device 206.

[0021] The controller unit 201 includes a CPU 202, a ROM 203, and a RAM 204. The CPU 202 is one or more CPUs that execute the processing of each processing unit in the cloud server 101. The ROM 203 stores various programs executed by the CPU 202 and data required by the programs. The RAM 204 is composed of SRAM, DRAM, etc., and stores program control variables, etc. Various work buffers are also stored in the RAM 204. The storage device 205 is composed of a hard disk, etc., and stores file data such as documents and images. The communication device 206 is an interface for connecting to a network. One connection method is a LAN.

[0022] The cloud server 101 accesses the Web content server 105 to acquire Web content, and executes a browser engine 103 program that interprets the Web content and generates a rendering result. The cloud server 101 also determines the element that is the target of the input operation based on the input operation received from the client device. The cloud server 101 also generates a rendering result according to the input operation. The cloud server 101 then transmits element information about the input operation target to the client device 102. The rendering result generated by the browser engine 103 is imaged using an imaging program 104 and transmitted to the client device 102. Each program is stored in the ROM 203 or the storage device 205, read into the RAM 204, and executed by the CPU 202. Communication with the Web content server 105 and the client device 102 can be performed via a communication device 206.

[0023] <Client device hardware configuration> 3 is a block diagram showing the hardware configuration of the client device 102 according to this embodiment. The client device 102 mainly includes a controller 301 that controls the entire client device 102, an operation device 305, a display device 306, a storage device 307, and a communication device 308.

[0024] The controller unit 301 includes a CPU 302, a ROM 303, and a RAM 304. The CPU 302 is one or more CPUs that execute the processing of each processing unit in the client device 102. The ROM 303 stores various programs executed by the CPU 302 and data required by the programs. The RAM 304 is composed of an SRAM, a DRAM, etc., and stores program control variables, etc. Various work buffers are also stored in the RAM 304.

[0025] The operation device 305 is made up of a keyboard, touch panel, mouse, etc. and is used by the user to perform various input operations. The display device 306 is used to display or notify the user using an LCD, LED display, etc. The storage device 307 is made up of a hard disk, etc. and stores file data such as documents and images. The communication device 308 is an interface for connecting to a network. One connection method is LAN.

[0026] The client device 102 executes various programs, such as programs related to UIs such as a URL input field and a web page display field, and programs related to sending and receiving communications. The programs are stored in the ROM 303 or the storage device 307, read into the RAM 304, and processed by the CPU 302.

[0027] A display device 306 displays UIs and images received from the cloud server 101. A URL specification and operations on a web page are accepted by an operation device 305. Communication with the cloud server 101 can be performed via a communication device 308.

[0028] <Client device functional configuration> 4 is a block diagram showing a schematic diagram of the software functions of the client device 102 in this embodiment. Each unit will be described. Note that each unit is controlled by a program stored in the ROM 303.

[0029] The communication unit 401 transmits and receives data to and from the cloud server 101. The communication unit 401 receives, from the cloud server 101, an image of the drawing result generated by the cloud server 101, and receives element information from the cloud server 101. The communication unit 401 also transmits to the cloud server 101 the content and coordinates of a user operation (such as a mouse input or a touch input) accepted by the client device 102.

[0030] The display unit 402 displays the image of the drawing result received by the communication unit 401, displays UIs such as URL input fields and dialogs generated by the client device 102, and displays UIs drawn by the UI drawing unit 404 described later.

[0031] The input unit 403 accepts user operations. It accepts mouse inputs and touch inputs (click, double click, mouse down, mouse up, mouse move, touch start, touch end, touch move, etc.) and coordinate inputs from the operation device 305, which is composed of a touch panel, a mouse, etc. The UI drawing unit 404 draws a user interface related to user operations on the Web content based on the detailed information of the element received by the communication unit 401. For example, it receives the rectangular coordinates of an element selected by a user operation as the detailed information of the element, and draws a rectangle surrounding the element on the image of the drawing result as the UI.

[0032] <Cloud server functional configuration> 5 is a block diagram showing a schematic diagram of the software functions of the cloud server 101 in this embodiment. Each unit will be described. Each unit is controlled by a program stored in the ROM 203.

[0033] The communication unit 501 transmits and receives data to and from the client device 102 and acquires web content from a web content server. The communication unit 501 also transmits images of the rendering results generated by the cloud server 101 and receives user operation information from the client device 102.

[0034] The content analysis unit 502 performs an analysis process on the Web content received from the Web content server 105. It performs syntax analysis of the HTML document, CSS, and JavaScript, and generates a DOM tree, CSSOM, and render tree, and a JavaScript AST (Abstract Syntax Tree).

[0035] The drawing unit 503 generates a drawing result based on the render tree generated by the content analysis unit 502 and the execution result of the JavaScript execution unit 504. The drawing unit 503 also manages the status of changes in the drawing result, and notifies other processing units of any changes in the drawing result.

[0036] The JavaScript execution unit 504 executes JavaScript using the AST generated by the content analysis unit 502 as input. The imaging processing unit 505 converts the drawing results generated by the drawing unit 503 into an image. The created image is sent to the client device 102 via the communication unit 501.

[0037] The input unit 506 executes the contents of mouse input and touch input, which are user operation information acquired from the communication unit 501, and the input of coordinates, as operations on the cloud server 101. The contents of mouse input and touch input include, for example, click, double click, mouse down, mouse up, mouse move, touch start, touch end, touch move, etc. This makes it possible to reflect user operations on the web page and dialog drawn by the drawing unit 503.

[0038] The element acquisition unit 507 acquires an element specified by a user operation received from the client device 102. The element acquisition unit 507 acquires an element specified by the coordinate information of the user operation based on the result of syntax analysis by the content analysis unit 502.

[0039] The identification unit 508 identifies the elements acquired by the element acquisition unit 507 and determines whether the element is a predetermined element. The predetermined element is, for example, an a element for describing a hyperlink. When this element is clicked, the link destination specified by the href attribute value becomes the final rendering result. Note that even if it is an a element, if the attribute value of href is "javascript:void(0)", it is determined as undefined, so no transition is made to the link destination specified in the href attribute. However, the rendering unit 503 generates a rendering result such that the user can know the element that has been clicked, such as surrounding the color and the range of the element with a rectangle, and then finally generates a rendering result indicating that "no transition has occurred", so it is determined that there are changes and transitions in the rendering result.

[0040] The transition identification unit 509 identifies the final rendering result in which a transition has occurred due to a user operation. For example, it is configured to be notified of a status that conveys a change in the rendering result from the rendering unit 503, ignores notifications of changes in the rendering result that occur within a certain period, and determines the notification after a certain period has elapsed as a notification of the generation of the final rendering result based on the user operation. Notifications for the rendering result that occur within a certain period include rendering notifications that display the clicked element so that it can be recognized, and since this notification is ignored, this rendering result is not imaged.

[0041] The imaging processing unit 505 images the rendering result identified by the transition identification unit 509 and transmits the image of the rendering result to the client device 101 via the communication unit 501.

[0042] When the element detail information acquisition unit 510 determines that the element is a predetermined element by the identification unit 508, it acquires the detailed information of the element. The detailed information of the element is, for example, information that defines a rectangle surrounding the element. The information that defines the rectangle is information including the coordinate information of the rectangle (x, y) of the upper left corner of the rectangle, the width width of the rectangle, and the height height of the rectangle, which is the coordinate information of the rectangle when the element is displayed as a UI. The acquired detailed information of the element is transmitted to the client device 101 via the communication unit 501.

[0043] <UI display example> 8(a) and 8(b) show examples of UIs displayed on a client device according to this embodiment. Fig. 8(a) shows a UI when a rendering result image of http: / / www.sample.org / received from the cloud server 101 is displayed on the client device 102. This is an example of rendering Web content in which four a elements (LinkAAA, LinkBBB, LinkCCC, LinkDDD) with linked URLs specified are arranged side by side.

[0044] FIG. 8(b) shows the UI when "LinkAAA" is clicked on the UI screen of FIG. 8(a). When "LinkAAA" in FIG. 8(a) is clicked, the client device 102 transmits information indicating "click," which is the operation content of the user operation, and information on the clicked coordinates to the cloud server 101. Thereafter, the client device 102 receives, as element information, coordinate information of a rectangle surrounding the clicked target element from the cloud server 101. Then, using the coordinate information of the rectangle, which are the upper left coordinates (x, y), the width (width) of the rectangle, and the height (height) of the rectangle, the client device 102 draws a rectangle on the drawing result image, which is the UI screen of FIG. 8(b). After this drawing, the client device 102 obtains the drawing result image of the link destination of "LinkAAA" from the cloud server 101 and displays it.

[0045] <Processing of Cloud Server 101> Next, the flow of processing of the cloud server 101 according to this embodiment will be described with reference to the flowchart shown in Fig. 6. Here, the description will start from the processing of obtaining Web content from the Web content server 105 via the communication unit 501, transmitting a rendering result image to the client device 102, and then receiving a user operation on the rendering result image.

[0046] In S601, the communication unit 501 receives a user operation from the client device 102. Then, the input unit 506 takes in the user operation, and the process proceeds to S602. Here, the user operation is, for example, the content of a mouse input or touch input and its coordinates.

[0047] In S602, the element acquisition unit 507 acquires the element that is the target of the user operation, and the process proceeds to S603. The browser engine 103 has a function to load and execute Javascript unique to the cloud server 101 into a web page acquired from the web content server 105. This function makes it possible to handle information about the web page that can be obtained by the unique Javascript, and for example, executes Javascript Document.elementFromPoint() to acquire the element at the specified coordinates.

[0048] In S603, the identification unit 508 identifies the element that is the target of the user operation and determines whether the element is a predetermined element. If it is determined that the element is a predetermined element (YES in S603), the process proceeds to S604. If it is determined that the element is not a predetermined element (NO in S603), the process proceeds to S605. For example, it determines whether the predetermined element is an a element.

[0049] In S604, the element detail information acquisition unit 510 acquires detail information about the element, and the process proceeds to S605. The detail information is, for example, information that defines a rectangle that surrounds the element. For example, Element.getBoundingClientRect() is executed by executing the aforementioned unique JavaScript. This acquires information including the top left coordinates (x, y) of the rectangle that surrounds the element, the width (width) of the rectangle, and the height (height) of the rectangle. The acquired detail information is sent to the client device 102 and used for UI rendering.

[0050] In S605, the communication unit 501 transmits the information to the client device 102, and the process proceeds to S606. For example, the information transmitted may be a URL of an image resulting from the user operation. If detailed information is acquired in S604, the acquired detailed information is also transmitted to the client device 102.

[0051] In S606, the transition identification unit 509 identifies the drawing result to be imaged and sent to the client device 102, and determines whether it is a transition image accompanying a page transition. That is, it determines whether the drawing result is a drawing result accompanying a page transition of Web content. If it is determined to be a transition image (YES in S606), proceed to S607. If it is determined not to be a transition image (NO in S606), proceed to S606. The image drawn last in response to the user operation is determined to be a transition image.

[0052] The browser engine 103 has a function of detecting and notifying changes in drawing. It determines that the drawing is the last drawing and that the drawing result has transitioned. For example, it determines that the last drawing is the last drawing within a certain period of time, or that there is no drawing within a certain period of time. It converts the drawing result in response to this drawing into an image and determines that it is the drawing result to be sent to the client device 102. This makes it possible to avoid converting the drawing result that encloses the element that is the target of the user operation in a rectangle (drawing result that should only be displayed for an instant).

[0053] In S607, the imaging processing unit 505 converts the designated drawing result into an image, and the process proceeds to S608. For example, the imaging processing unit 505 converts the drawing result determined by the transition identification unit 509 to be a transitioned drawing result into an image file such as PNG or JPEG.

[0054] In S608, the communication unit 501 determines whether or not an image acquisition request has been received from the client device 102 (request reception). If an image acquisition request has been received (S608-YES), the process proceeds to S609. If an image acquisition request has not been received (S608-NO), the process proceeds to S608. For example, the image acquisition request specifies the URL of the rendering result image obtained in S605, and the image that the client device 102 wants to acquire can be determined in subsequent processing.

[0055] In S609, the imaging processing unit 505 determines whether the imaging process of the image specified by the URL has been completed. If it has been completed (S609-YES), the process proceeds to S611. If it has not been completed (S609-NO), the process proceeds to S610.

[0056] In S610, the communication unit 501 returns status information indicating that image generation is not complete to the client device 102. Thereafter, the process returns to S608. In S611, the communication unit 501 transmits the rendering result image generated by the imaging processing unit 505 to the client device 102, and ends the process (result transmission). Note that the communication unit 501 may also transmit status information indicating that image generation is complete together with the rendering result image.

[0057] <Processing of the client device 102> Next, the processing flow of the client device 102 of this embodiment will be described with reference to the flowchart shown in Fig. 7. In S701, the communication unit 401 transmits information about a user operation on the drawing result image accepted by the input unit 403 to the cloud server 101, and the process proceeds to S702. The information about the user operation is, for example, the content of a mouse input or touch input and the input coordinates.

[0058] In S702, the communication unit 401 receives a response from the cloud server 101 and determines whether the response includes element detail information. For example, it determines whether the response includes element detail information in addition to the URL of the rendering result image. If the response includes element detail information (S702-YES), proceed to S703. If the response does not include element detail information (S702-NO), proceed to S704.

[0059] In S703, the UI drawing unit 404 draws a UI on the displayed drawing result image using the element detailed information, and proceeds to S704. For example, if the user clicks on "Link AAA" on the UI displaying the drawing result image of FIG. 8(a), the upper left coordinates (x, y) of the rectangle surrounding the a element of "Link AAA", the width of the rectangle (width), and the height of the rectangle (height) are received as element detailed information. Using this element detailed information, a rectangle surrounding "Link AAA" is displayed as shown in FIG. 8(b).

[0060] In S704, the communication unit 401 transmits a request to the cloud server 101 requesting a rendering result image, and the process proceeds to S705 (request transmission). The process of this step is a process of requesting a transition image in response to a user operation. For example, by specifying the URL received in S702 in this request, the image to be acquired can be transmitted to the cloud server 101.

[0061] In S705, the communication unit 401 determines whether or not a rendering result image has been acquired from the cloud server 101 (receives a result). It determines whether or not the rendering result image is included in the response to the request sent in S704. If the response includes status information for the generation of the rendering result image, the status information can also be acquired. If the rendering result image has been acquired (YES in S705), the process proceeds to S706. If the rendering result image has not been acquired (NO in S705), the process returns to S704.

[0062] In S706, the display unit 402 displays the acquired rendering result image. For example, if the user selects (clicks) "LinkAAA" shown in FIG. 8(a), the client device 102 draws a rectangle surrounding "LinkAAA" as shown in FIG. 8(b). After that, the image transitioned to the destination specified by the href attribute of the a element of "LinkAAA" is received as the rendering result image and displayed. Then, the processing ends.

[0063] As described above, in this embodiment, after the UI (rectangle) is drawn by the UI drawing unit 404, the drawing result received from the cloud server 101 is displayed. According to this embodiment, the cloud server 101 can delegate some drawing processes to the client device 102 by transmitting the detailed element information of the user operation target to the client device 102, and can omit the imaging process of some drawing results. According to this embodiment, by generating the drawing result for displaying some UI changes on the client device side, the UI changes can be displayed on the client device without delay, and the cloud server can shift to the generation process of the drawing result of the transition destination more quickly.

[0064] (Embodiment 2) In Embodiment 2, when the element content and the transition destination information are received as the detailed element information, an example of the UI drawn by the UI drawing unit 404 of the client device 102 will be described. Note that the same reference numerals are used for the same processes as in Embodiment 1, and the description thereof will be omitted.

[0065] <UI Display Example> FIG. 9 and FIG. 10 show UI display examples when the user selects (clicks) "LinkAAA" in FIG. 8(a). These are examples of the UI drawn by the UI drawing unit 404 in S703 when "LinkAAA", which is the element content of "LinkAAA", and the transition destination URL "http: / / aaa" are received in S702. By showing the element content and the transition destination URL, the selected (clicked) element can be conveyed to the user

[0066] FIG. 9 shows an example in which a balloon containing the element content "LinkAAA" and the transition destination URL "http: / / aaa" is displayed based on the position operated by the user. FIG. 10 shows an example in which the element content "LinkAAA" and the transition destination URL "http: / / aaa" are displayed as detailed information in a dialog. In addition to displaying the element detailed information, the dialog is provided with a "Cancel" button for canceling access to the transition destination and an "OK" button for closing the dialog. In this way, in addition to displaying the element detailed information, a UI for executing another process may also be displayed. If the user makes an incorrect selection (click), the user can click the "Cancel" button to request cancellation of the transition. When the "Cancel" button is clicked, a rendering cancel request specifying the URL of the rendering result image is sent to the cloud server 101. If the cloud server 101 is in the middle of generating a rendering result for the specified URL, the generation process can be stopped.

[0067] For example, if the cloud server 101 has not yet drawn the transition destination in the processing of S606, it can receive a "cancel" request from the client device 102 and cancel the transition. The cloud server 101 returns a response to the "cancel" request indicating whether or not the request was cancelled. The client device 102 that has received the response displays on the UI whether or not the request was cancelled after closing the dialog, and if the request was not cancelled, it displays the image of the drawing result that will be received thereafter. On the other hand, if the request was cancelled, the image of the drawing result is not received or displayed.

[0068] (Embodiment 3) In the first embodiment, an example in which the a element is the predetermined element has been described. In this embodiment, an example in which a submit-type input element is the predetermined element will be described.

[0069] Fig. 11 shows an example of HTML notation for a submit-type input element according to this embodiment. Fig. 11 shows that the transition destination when the Submit button is clicked is specified by the action element in the action attribute of the form element, which is the parent element of the input element.

[0070] <Processing of Cloud Server 101> The processing of the cloud server 101 according to this embodiment will be described, focusing on the differences from FIG. 6. In S603, the identification unit 508 identifies the element to be operated by the user and determines whether the element is a predetermined element. In this embodiment, the predetermined element is a submit-type input element. If the element to be operated by the user is a submit-type input, the process proceeds to S604. If the element to be operated by the user is not a submit-type input, the process proceeds to S605.

[0071] In S604, the element detailed information acquisition unit 510 acquires the transition destination of the submit element as detailed information about the element. The transition destination of the submit element can also be specified by the formaction attribute of the input element, but in the case of Figure 11, formaction is not specified for the input element. In this case, / submit_page.php, which is the destination specified by the action attribute value of the parent form element, is accessed, and this value is returned as detailed information about the element. In this way, detailed information may be acquired not from the element itself that is the target of user operation, but from an element related to that element. Detailed information can also be changed for each specified element.

[0072] [Variations] In the above-described embodiments, the UI drawing unit 404 draws a rectangle surrounding an element or a speech bubble or dialog box including the element content and the destination URL, but the present invention is not limited to these examples. If it is possible to recognize that the user has touched or clicked a link, the link may be highlighted in other ways. For example, the link text may be colored or bolded.

[0073] The disclosure of this specification includes the following information processing system, information processing device, control method for an information processing device, and program. (Item 1) An information processing system in which a first information processing device processes web content and transmits a rendering result obtained to a second information processing device for display, The first information processing device a receiving means for receiving information regarding a user operation on the web content from the second information processing device; an element acquisition means for acquiring an element that is a target of the user operation; an information acquisition means for acquiring detailed information of the element when the element is a predetermined element; a transmitting means for transmitting the detailed information to the second information processing device, The second information processing device receiving means for receiving the detailed information; a drawing means for drawing a user interface relating to the user operation on the web content based on the detailed information; An information processing system comprising: (Item 2) The first information processing device a determination means for determining whether the rendering result is a rendering result involving a page transition of the Web content; an imaging means for imaging the drawing result determined to involve the page transition; a request receiving means for receiving an image acquisition request from the second information processing device; a result transmission means for transmitting the drawing result imaged by the imaging means to the second information processing device in response to receiving the image acquisition request; 2. The information processing system according to item 1, further comprising: (Item 3) The second information processing device a request sending means for sending the image acquisition request to the first information processing device; a result receiving means for receiving the drawing result from the first information processing device; a display means for displaying the drawing result received by the result receiving means after the drawing is performed by the drawing means; 3. The information processing system according to item 2, further comprising: (Item 4) 4. The information processing system according to any one of items 1 to 3, wherein the detailed information includes information defining a rectangle surrounding the element. (Item 5) 5. The information processing system according to item 4, wherein the information defining the rectangle includes information on the top left coordinate of the rectangle, the width of the rectangle, and the height of the rectangle. (Item 6) 6. The information processing system according to item 4 or 5, wherein the drawing means of the second information processing device draws the rectangle surrounding the element. (Item 7) 4. The information processing system according to any one of items 1 to 3, wherein the detailed information includes the element content of the element and a transition destination URL. (Item 8) 8. The information processing system according to item 7, wherein the drawing means of the second information processing device draws a speech bubble including the element content and the URL of the transition destination. (Item 9) 8. The information processing system according to item 7, wherein the drawing means of the second information processing device draws a dialog including the element content and the URL of the transition destination. (Item 10) 10. The information processing system according to item 9, wherein the dialog includes a button for canceling the transition to the transition destination and a button for closing the dialog. (Item 11) 11. The information processing system according to any one of items 1 to 10, wherein the predetermined element is an a element for describing a hyperlink. (Item 12) 12. The information processing system according to any one of items 1 to 11, wherein the predetermined element is a submit-type input element. (Item 13) 13. The information processing system according to any one of items 1 to 12, wherein the information about the user operation includes the contents of a mouse input or a touch input and input coordinates. (Item 14) the first information processing device is a cloud server, The second information processing device is a client device. 14. The information processing system according to any one of items 1 to 13, (Item 15) An information processing device that processes web content and transmits a drawing result obtained to another information processing device for display, a receiving means for receiving information regarding a user operation on the web content from the other information processing device; an element acquisition means for acquiring an element that is a target of the user operation; an information acquisition means for acquiring detailed information of the element when the element is a predetermined element; a transmitting means for transmitting the detailed information to the other information processing device; An information processing device comprising: (Item 16) An information processing device that receives and displays a rendering result obtained by processing web content in another information processing device from the other information processing device, a transmitting means for transmitting information relating to a user operation in the information processing device to the other information processing device; a receiving means for receiving a response from the other information processing device; a rendering means for rendering a user interface relating to the user operation on the web content based on the detailed information when the response includes detailed information about the element that is the target of the user operation; An information processing device comprising: (Item 17) A control method for an information processing device that processes web content and transmits a rendering result to another information processing device for display, a receiving step of receiving information regarding a user operation on the web content from the other information processing device; an element acquisition step of acquiring an element that is a target of the user operation; an information acquisition step of acquiring detailed information about the element if the element is a predetermined element; a transmitting step of transmitting the detailed information to the other information processing device; 1. A method for controlling an information processing device, comprising: (Item 18) A control method for an information processing device that receives and displays a rendering result obtained by processing web content in another information processing device from the other information processing device, a transmitting step of transmitting information regarding a user operation in the information processing device to the other information processing device; a receiving step of receiving a response from the other information processing device; a rendering step of rendering a user interface relating to the user operation on the web content based on the detailed information when the response includes detailed information about the element that is the target of the user operation; 1. A method for controlling an information processing device, comprising: (Item 19) 19. A program for causing a computer to execute the method for controlling an information processing device according to item 17 or 18.

[0074] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0076] 100: Information processing system, 101: Cloud server, 102: Client device, 103: Browser engine, 104: Imaging program, 105: Web content server, 401: Communication unit, 402: Display unit, 403: Input unit, 404: UI drawing unit, 501: Communication unit, 502: Content analysis unit, 503: Drawing unit, 504: JavaScript execution unit, 505: Imaging processing unit, 506: Input unit, 507: Element acquisition unit, 508: Identification unit, 509: Transition identification unit, 510: Element detailed information acquisition unit

Claims

1. An information processing system in which a first information processing device processes web content, and transmits a rendering result obtained from the processing to a second information processing device for display, The first information processing device a receiving means for receiving information relating to a user operation on the web content from the second information processing device; an element acquisition means for acquiring an element that is a target of the user operation; an information acquisition means for acquiring detailed information of the element when the element is a predetermined element; a transmitting means for transmitting the detailed information to the second information processing device, The second information processing device receiving means for receiving the detailed information; a drawing means for drawing a user interface relating to the user operation on the web content based on the detailed information; An information processing system comprising:

2. The first information processing device a determining means for determining whether the rendering result is a rendering result involving a page transition of the web content; an imaging means for imaging the drawing result determined to involve the page transition; a request receiving means for receiving an image acquisition request from the second information processing device; a result transmitting means for transmitting the drawing result imaged by the imaging means to the second information processing device in response to receiving the image acquisition request; The information processing system according to claim 1 , further comprising:

3. The second information processing device a request sending means for sending the image acquisition request to the first information processing device; a result receiving means for receiving the drawing result from the first information processing device; a display means for displaying the drawing result received by the result receiving means after the drawing is performed by the drawing means; The information processing system according to claim 2, further comprising:

4. 4. The information processing system according to claim 1, wherein the detailed information includes information defining a rectangle surrounding the element.

5. 5. The information processing system according to claim 4, wherein the information defining the rectangle includes information on the upper left coordinate of the rectangle, the width of the rectangle, and the height of the rectangle.

6. 5. The information processing system according to claim 4, wherein the drawing means of the second information processing device draws the rectangle surrounding the element.

7. 2. The information processing system according to claim 1, wherein the detailed information includes the element content of the element and a URL of a transition destination.

8. 8. The information processing system according to claim 7, wherein the drawing means of the second information processing apparatus draws a speech bubble including the element content and the URL of the transition destination.

9. 8. The information processing system according to claim 7, wherein the drawing means of the second information processing apparatus draws a dialog including the element content and the URL of the transition destination.

10. 10. The information processing system according to claim 9, wherein the dialog includes a button for canceling the transition to the transition destination and a button for closing the dialog.

11. 2. The information processing system according to claim 1, wherein the predetermined element is an a element for describing a hyperlink.

12. 2. The information processing system according to claim 1, wherein the predetermined element is a submit-type input element.

13. 2. The information processing system according to claim 1, wherein the information about the user operation includes the contents and coordinates of a mouse input or a touch input.

14. the first information processing device is a cloud server, The second information processing device is a client device.

2. The information processing system according to claim 1, wherein:

15. An information processing device that processes web content, transmits a drawing result obtained to another information processing device, and displays the drawing result, a receiving means for receiving information relating to a user operation on the web content from the other information processing device; an element acquisition means for acquiring an element that is a target of the user operation; an information acquisition means for acquiring detailed information of the element when the element is a predetermined element; a transmitting means for transmitting the detailed information to the other information processing device; An information processing device comprising:

16. An information processing device that receives and displays a rendering result obtained by processing web content in another information processing device from the other information processing device, a transmitting means for transmitting information relating to a user operation in the information processing device to the other information processing device; a receiving means for receiving a response from the other information processing device; a rendering means for rendering a user interface relating to the user operation on the web content based on the detailed information when the response includes detailed information about the element that is the target of the user operation; An information processing device comprising:

17. A control method for an information processing device that processes web content, transmits a drawing result obtained from the processing to another information processing device, and displays the drawing result, comprising: a receiving step of receiving information regarding a user operation on the web content from the other information processing device; an element acquisition step of acquiring an element that is a target of the user operation; an information acquisition step of acquiring detailed information about the element if the element is a predetermined element; a transmitting step of transmitting the detailed information to the other information processing device; 1. A method for controlling an information processing device, comprising:

18. A control method for an information processing device that receives and displays a rendering result obtained by processing web content in another information processing device from the other information processing device, comprising: a transmitting step of transmitting information regarding a user operation in the information processing device to the other information processing device; a receiving step of receiving a response from the other information processing device; a rendering step of rendering a user interface relating to the user operation on the web content based on the detailed information when the response includes detailed information of the element that is the target of the user operation; 1. A method for controlling an information processing device, comprising:

19. 19. A program for causing a computer to execute the method for controlling an information processing device according to claim 17 or 18.

Images