Information processing system, information processing apparatus, communication apparatus, control method, and program

Abstract

To provide a technique for shortening the time required for displaying a keyboard in response to a user touch.SOLUTION: An information processing system that includes a communication apparatus and an information processing apparatus for providing a rendering result of a web page to the communication apparatus based on a request from the communication apparatus, the information processing system comprises: analysis means for analyzing the web page to determine a text input area; and transmitting means for transmitting the rendering result and information of the text input area to the communication apparatus. The communication apparatus comprises receiving means for receiving information about the text input area in the rendering result from the information processing apparatus, receiving means for receiving an input operation from a user, determining means for determining whether or not an operation position of the input operation is included in the text input area, and activation means for activating a keyboard if the operation position is included in the text input area.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, a communication device, a control method, and a program, and in particular to UI (User Interface) processing. [Background technology]

[0002] Conventionally, a technique for rendering a screen using HTML (https: / / html.spec.whatwg.org) is known. An application that processes HTML is called a browser, and renders images according to the HTML specifications.

[0003] In recent years, it has become possible to run apps on cloud services, allowing users to send their operation information from their local devices to apps on the cloud service, and for the devices to receive and display the results of the app's processing to the user.

[0004] The cloud browser is a browser that runs in the application execution environment of a cloud service. The cloud browser performs drawing in response to requests from a device and saves the drawing results in a database of the same cloud service. The device retrieves the drawing results from the database and displays them on the device screen.

[0005] Here, if the HTML contains an input form, when the user touches the input form on the screen, the device sends a touch event to the cloud browser. This is because the device only has the drawing result and does not have information about the input form. The cloud browser processes the touch event and returns information to the device about whether the touched element is an input form. If it is an input form, the device displays an on-screen keyboard on the screen.

[0006] However, when determining whether the location touched by the user is an input form, time is required for communication between the device and the cloud browser, which causes a delay between the user's touch and the display of the keyboard.In response to this, Patent Document 1 discloses a technology that uses OCR (Optical Character Recognition) on an image to determine where input is possible. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2012-059248 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, the technology described in Patent Document 1 uses OCR, and has problems in that it cannot reflect the accurate position and size of the input form, and furthermore, OCR processing takes time.

[0009] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that shortens the time it takes for a keyboard to be displayed in response to a user's touch when using a cloud browser. [Means for solving the problem]

[0010] An information processing system according to one aspect of the present invention that achieves the above object comprises: An information processing system including a communication device and an information processing device that provides a rendering result of a web page to the communication device based on a request from the communication device, The information processing device includes: analysis means for analyzing the web page to determine text input areas; a transmitting means for transmitting the drawing result and information of the text input area to the communication device, The communication device includes: a receiving means for receiving information on the text input area in the drawing result from the information processing device; A receiving means for receiving an input operation from a user; a determination means for determining whether an operation position of the input operation is included in the text input area; an activation means for activating a keyboard when the operation position is included in the text input area; The present invention is characterized by comprising: Effect of the Invention

[0011] According to the present invention, when a cloud browser is used, it is possible to reduce the time required for a keyboard to be displayed in response to a user's touch. [Brief description of the drawings]

[0012] [Figure 1] FIG. 4 is a schematic diagram illustrating an example of a configuration of a cloud server according to an embodiment. [Diagram 2] 10 is a flowchart illustrating an example of a process of a cloud server according to an embodiment. [Diagram 3] FIG. 1 is a schematic diagram showing an example of a configuration of a device according to an embodiment. [Figure 4] 10 is a flowchart showing an example of processing of a device according to an embodiment. [Diagram 5] FIG. 11 is a diagram showing an example of HTML acquired by a cloud browser according to an embodiment. [Figure 6] FIG. 11 is a diagram showing an example of a rendering result generated by a cloud browser according to an embodiment. [Figure 7] 11A and 11B are diagrams showing examples of drawing results generated by a cloud browser after text input according to an embodiment. [Figure 8] FIG. 2 is a diagram showing an example of an on-screen keyboard according to an embodiment. [Figure 9] FIG. 4 is a diagram showing an example of a processing sequence in an information processing system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] In this embodiment, an example is described in which a cloud server sends information regarding where the text input area is located in the drawing result to the device along with the drawing result, thereby enabling the device to quickly launch the keyboard when the text input area is touched.

[0015] <Configuration> 1 is a configuration diagram showing an overview of a cloud server according to this embodiment. Cloud server 101 functions as an information processing device, and includes a CPU 108, ROM 102, RAM 103, an external storage unit 104, an input unit 105, and a network connection interface 107. A virtual machine is stored in external storage unit 104 of cloud server 101, and is loaded into RAM 103 to start up the virtual machine.

[0016] The cloud browser runs on a virtual machine. It accesses an external web server based on a URL specified via the network connection interface 107, acquires the content that constitutes a web page, and outputs the result of processing the content to the RAM 103 as a rendering result.

[0017] CPU 108 is a system control unit composed of one or more CPUs, and controls the entire device. ROM 102 stores the CPU control program and various fixed data. RAM 103 is composed of SRAM, DRAM, etc., and stores program control variables, etc. Various setting parameters and various work buffers are also stored in RAM 103.

[0018] The external storage unit 104 is composed of a hard disk or the like, and stores file data such as documents and images. The input unit 105 is composed of a keyboard, a touch panel, and the like, and is used by an operator to perform various input operations. The network connection interface 107 is an interface for connecting to a network. Connection methods include a LAN (Local Area Network) and a USB (Universal Serial Bus). Note that the configuration shown in FIG. 1 shows a part of a cloud server that functions as the drawing device according to this embodiment. Also, when this embodiment is realized by software or the like, each block in FIG. 1 does not necessarily have to be included in the device.

[0019] <Equipment configuration> 3 is a block diagram showing an overview of a device according to this embodiment. A device 301 functions as a communication device, and includes a CPU 308, a ROM 302, a RAM 303, an external storage unit 304, an input unit 305, a display unit 306, a network connection interface 307, a scanner unit 309, and a printer unit 310.

[0020] CPU 308 is a system control unit that is made up of one or more CPUs and controls the entire device. ROM 302 stores the CPU control programs and various fixed data. RAM 303 is made up of SRAM, DRAM, etc., and stores program control variables, etc. Various setting parameters and various work buffers are also stored in RAM 303.

[0021] The external storage unit 304 is composed of a hard disk or the like, and stores file data such as documents and images. The input unit 305 is composed of a keyboard, a touch panel, and the like, and is used by an operator to perform various input operations. The display unit 306 is used to display a UI to the user. The network connection interface 307 is an interface for connecting to a network. Connection methods include LAN and USB. The scanner unit 309 scans paper documents. The printing unit 310 prints documents, images, and the like. The configuration shown in FIG. 3 shows a part of the device according to this embodiment. Also, when this embodiment is realized by software or the like, each block in FIG. 3 does not necessarily have to be included in the device.

[0022] <Cloud server processing> The operation of the cloud server 101 according to this embodiment will be described below with reference to the flowchart in FIG.

[0023] In S201, the cloud server 101 receives a URL via the network connection interface 107. The received URL information is stored in the RAM 103.

[0024] In S202, the cloud server 101 acquires HTML from a Web server that provides a Web page based on the received URL via the network connection interface 107. The HTML may be, for example, as shown in FIG.

[0025] In S203, the cloud server 101 analyzes the acquired HTML. If the HTML refers to external CSS or JavaScript, the cloud server 101 also acquires them via the network connection interface 107 and analyzes them. The analysis result includes information such as whether or not text input is required for the HTML. <input> The analysis result is stored in the RAM 103. The analysis result includes information on whether the element is included.

[0026] In S204, the cloud server 101 renders HTML based on the analysis result. The rendering result is, for example, as shown in FIG. 6. In this embodiment, the screen resolution of the display unit 306 is fixed at 600×480, but may be variable. If variable, the screen resolution may be transmitted together with the URL in S401 of FIG. 4, which will be described later. The rendering result is stored in the RAM 103 as an image file. The image file may be JPEG or PNG.

[0027] In S205, the cloud server 101 determines whether or not the drawing result includes a text input area. If it does, the process proceeds to S206. On the other hand, if it does not, the process proceeds to S211. The text input area is defined as follows in HTML: <input> The analysis result stored in the RAM 103 is used to determine whether or not a text input area is included.

[0028] In S206, the cloud server 101 calculates where the text input area is located within the drawing result. In this embodiment, the HTML includes a first text input area for inputting a user name and a second text input area for inputting a password. In this embodiment, the text input area is a rectangle defined by arbitrary coordinates, width, and height, and is, for example, a rectangle with the upper left corner of the drawing result as its origin, and is defined by the upper left coordinates, width, and height of the rectangle. For the calculation, getBoundingClientRect() of the DOM API or the like can be used.

[0029] The first text input area is a rectangle with an x ​​coordinate of 8, a y coordinate of 8, a width of 180, and a height of 20. The second text input area is a rectangle with an x ​​coordinate of 8, a y coordinate of 28, a width of 180, and a height of 20. The text input area may be enlarged, reduced, rotated, or otherwise transformed by CSS or JavaScript. The text input area information may include the following: <input>It may also include information such as the type attribute and placeholder attribute, which are attributes of the element. The placeholder is the string that is displayed when no input is made. The text input area information may also include <input> Style information may include CSS such as colors and border widths to be applied to elements.

[0030] In S207, cloud server 101 transmits the drawing result and the text input area information to device 301 via network connection interface 107. The data may be compressed when transmitted. Alternatively, the drawing result and the text input area information may be stored on the network, and the URL may be transmitted to device 301.

[0031] In S208, the cloud server 101 determines whether or not a touch event and a text input event have been received from the device 301 via the network connection interface 107. If so, the process proceeds to S209. If not, the process ends.

[0032] In S209, the cloud server 101 processes the received event. In this embodiment, since a touch event has occurred in the first text input area, the first text input area enters a text input state. Furthermore, the cloud server 101 processes the text input event. In this embodiment, since the user has entered a username, the username is entered in the first text input area. The cloud server 101 generates a drawing result reflecting the new HTML state and saves it in the RAM 103. The drawing result is, for example, as shown in FIG. 7. In the example of FIG. 7, bob has been entered as the username.

[0033] In S210, cloud server 101 transmits the drawing result to device 301 via network connection interface 107. In S211, cloud server 101 transmits the drawing result to device 301 via network connection interface 107. The above is the series of processes in FIG.

[0034] <Device Processing> The operation of the device 301 according to this embodiment will be described with reference to the flowchart in Fig. 4. First, in step 401, the device 301 transmits a URL via the network connection interface 307. The communication protocol may be HTTP or HTTPS.

[0035] In S402, the device 301 receives data via the network connection interface 307. The data includes a drawing result, and may further include text input area information. The received data is stored in the RAM 303.

[0036] In S403, the device 301 displays the drawing result included in the received data on the display unit 306. In this embodiment, the screen resolution of the display unit 306 is 600×480, so the drawing result is displayed as is, but if the screen resolution does not match, the drawing result may be scaled. In that case, the same scale is also applied to the text input area.

[0037] In S404, the device 301 determines whether the received data includes text input area information. If it does, the process proceeds to S405. If it does not, the process ends.

[0038] In S405, the device 301 judges whether or not the user has touched the drawing result. If the drawing result has been touched, the process proceeds to S406. On the other hand, if the drawing result has not been touched, the process ends. The case where the drawing result has not been touched may be, for example, a case where a certain period of time has passed without touching. The input unit 305 stores the touched coordinates in the RAM 303 as coordinates with the upper left corner of the drawing result as the origin. In this embodiment, the input unit 305 is a touch screen, and it is judged whether or not the user has touched the touch screen with a finger. The input unit 305 may be a pen input or a mouse input.

[0039] In S406, the device 301 determines whether the user has touched the text input area. If the touched coordinates are included in the rectangle of the text input area, the process proceeds to S407. On the other hand, if the touched coordinates are not included in the rectangle of the text input area, the process ends.

[0040] In S407, the device 301 accepts text input from the user. In this embodiment, an on-screen keyboard is displayed on the display unit 306, and the user inputs text by touching the touch screen that is the input unit 305. The user may also input text by typing on a hardware keyboard connected to the device 301. The user may also input text by voice recognition.

[0041] In addition, if the text input area contains style information such as CSS, the style information may be reflected in the on-screen keyboard. For example, <input> When the placeholder attribute of an element is reflected in the input area of ​​the on-screen keyboard, the display on display unit 306 will look like that shown in Fig. 8. In this way, by adding HTML attribute and CSS information applied to the input form to the information in the text input area and reproducing it on device 301, it is possible to improve the consistency of the display for the user.

[0042] In S408, the device 301 transmits the touch event information and the text input event information to the cloud server 101 via the network connection interface 307. The touch event information includes information on the coordinates of the touch by the user. The text input event information includes information on the text input by the user using the on-screen keyboard.

[0043] In S409, the device 301 receives the rendering result via the network connection interface 307, and displays it on the display unit 306. The above is the series of processes in FIG.

[0044] <Processing sequence> Next, a processing sequence in the information processing system including the above-mentioned cloud server 101 (information processing device) and device 301 (communication device) will be described with reference to the sequence diagram of FIG.

[0045] In F901, the user inputs URL information by operating the device 301. In F902, the device 301 transmits the input URL information to the cloud server 101. In F903, the cloud server 101 acquires and analyzes HTML via the network connection interface 107 based on the received URL.

[0046] In F904, the cloud server 101 draws HTML based on the analysis result. Then, the cloud server 101 determines whether the drawing result includes a text input area, and if so, calculates where the text input area is located within the drawing result. In F905, the cloud server 101 transmits the drawing result and (if the drawing result includes a text input area) text input area information to the device 301.

[0047] In F906, the device 301 displays the drawing result included in the received data on the display unit 306. In F907, the user performs a touch operation by operating the device 301. In F908, the device 301 determines whether the touch position is in a text input area.

[0048] In F909, if the touch position is in a text input area, the device 301 displays an on-screen keyboard on the display unit 306. In F910, the user operates the device 301 to input text by touching the touch screen that is the input unit 305.

[0049] In F911, the device 301 transmits touch event information and text input event information to the cloud server 101. The touch event information includes information on the coordinates of the touch by the user. The text input event information includes information on the text input by the user using the on-screen keyboard.

[0050] In F912, the cloud server 101 processes the event received from the device 301 and generates a drawing result. In F913, the cloud server 101 transmits the generated drawing result, which is the event processing result, to the device 301. In F914, the device 301 displays the drawing result received from the cloud server 101 on the display unit 306. This completes the processing sequence in FIG. 9.

[0051] As described above, in this embodiment, the cloud server acquires information regarding where the text input area is located in the drawing result, and transmits the information to the device together with the drawing result. By adding the information on the text input area to the drawing result in this way, it is not necessary to perform new communication in response to the user's touch operation, and the device can quickly launch the keyboard when the text input area is touched. In other words, when using a cloud browser, it is possible to reduce the time required to display the keyboard in response to a user's touch (to instantly launch the keyboard).

[0052] [Variations] Examples of storage media that can be used to supply the drawing program code according to this embodiment include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, DVDs, etc.

[0053] The functions of the above-mentioned embodiments may be realized by executing the drawing program code read by the computer. Furthermore, an operating system (OS) running on the computer may perform part or all of the actual processing based on the instructions of the drawing program code. The functions of the above-mentioned embodiments may be realized by this processing.

[0054] Furthermore, the drawing program code read from the storage medium may be written to a memory provided in a function expansion board inserted into a computer or a function expansion unit connected to the computer. Then, based on the instructions of the drawing program code, a CPU or the like provided in the function expansion board or function expansion unit may perform part or all of the actual processing, and the functions of the above-mentioned embodiments may be realized by the processing.

[0055] The disclosure of this specification includes the following information processing system, information processing device, communication device, control method, and program.

[0056] (Item 1) An information processing system including a communication device and an information processing device that provides a rendering result of a web page to the communication device based on a request from the communication device, The information processing device includes: analysis means for analyzing the web page to determine text input areas; a transmitting means for transmitting the drawing result and information of the text input area to the communication device, The communication device includes: a receiving means for receiving information on the text input area in the drawing result from the information processing device; A receiving means for receiving an input operation from a user; a determination means for determining whether an operation position of the input operation is included in the text input area; an activation means for activating a keyboard when the operation position is included in the text input area; An information processing system comprising:

[0057] (Item 2) The information processing device includes: 2. The information processing system according to item 1, further comprising a receiving means for receiving, from the communication device, input data inputted into the text input area of ​​the communication device.

[0058] (Item 3) the information about the text entry area includes style information to be applied to the text entry area; 3. The information processing system according to item 1 or 2, wherein the activation means reflects the style information on the keyboard.

[0059] (Item 4) 4. The information processing system according to any one of items 1 to 3, wherein the text input area is a rectangle defined by arbitrary coordinates, a width, and a height.

[0060] (Item 5) 5. The information processing system according to any one of items 1 to 4, wherein the input operation is a touch operation.

[0061] (Item 6) 6. The information processing system according to any one of items 1 to 5, wherein the analysis means determines the text input area by acquiring HTML from a web server that provides the web page and analyzing the HTML.

[0062] (Item 7) 1. An information processing device that provides a rendering result of a web page to a communication device based on a request from the communication device, analysis means for analyzing the web page to determine text input areas; a transmitting means for transmitting the drawing result and information of the text input area to the communication device; An information processing device comprising:

[0063] (Item 8) A communication device that displays a rendering result of a web page received from an information processing device, a receiving means for receiving information on a text input area in the drawing result from the information processing device; A receiving means for receiving an input operation from a user; a determination means for determining whether an operation position of the input operation is included in the text input area; an activation means for activating a keyboard when the operation position is included in the text input area; A communication device comprising:

[0064] (Item 9) 1. A method for controlling an information processing device that provides a rendering result of a web page to a communication device based on a request from the communication device, comprising: an analyzing step of analyzing the web page to determine text input areas; a transmitting step of transmitting the drawing result and information of the text input area to the communication device; 13. A method for controlling an information processing apparatus comprising the steps of:

[0065] (Item 10) A method for controlling a communication device that displays a rendering result of a web page received from an information processing device, comprising: a receiving step of receiving information on a text input area in the drawing result from the information processing device; a receiving step of receiving an input operation from a user; a determination step of determining whether an operation position of the input operation is included in the text input area; an activation step of activating a keyboard when the operation position is included in the text input area; 13. A method for controlling a communication device, comprising:

[0066] (Item 11) A program for causing a computer to execute the control method according to item 9 or 10.

[0067] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0068] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0069] 101: cloud server, 102: ROM, 103: RAM, 104: external storage unit, 105: input unit, 107: network connection interface, 108: CPU, 301: device, 302: ROM, 303: RAM, 304: external storage unit, 305: input unit, 306: display unit, 307: network connection interface, 308: CPU, 309: scanner unit, 310: printing unit

Claims

1. An information processing system including a communication device and an information processing device that provides a rendering result of a web page to the communication device based on a request from the communication device, The information processing device includes: analysis means for analyzing the web page to determine a text input area; a transmitting means for transmitting the drawing result and information of the text input area to the communication device, The communication device includes: a receiving means for receiving information on the text input area in the drawing result from the information processing device; A receiving means for receiving an input operation from a user; a determination means for determining whether an operation position of the input operation is included in the text input area; an activation means for activating a keyboard when the operation position is included in the text input area; An information processing system comprising:

2. The information processing device includes:

2. The information processing system according to claim 1, further comprising a receiving means for receiving, from said communication device, input data inputted into said text input area in said communication device.

3. the information about the text entry area includes style information to be applied to the text entry area; 2. The information processing system according to claim 1, wherein said activation means reflects said style information on said keyboard.

4. 2. The information processing system according to claim 1, wherein the text input area is a rectangle defined by arbitrary coordinates, a width, and a height.

5. The information processing system according to claim 1 , wherein the input operation is a touch operation.

6. 2. The information processing system according to claim 1, wherein said analyzing means determines said text input area by acquiring HTML from a Web server that provides said Web page and analyzing it.

7. 1. An information processing device that provides a rendering result of a web page to a communication device based on a request from the communication device, analysis means for analyzing the web page to determine a text input area; a transmitting means for transmitting the drawing result and information of the text input area to the communication device; An information processing device comprising:

8. A communication device that displays a rendering result of a web page received from an information processing device, a receiving means for receiving information on a text input area in the drawing result from the information processing device; A receiving means for receiving an input operation from a user; a determination means for determining whether an operation position of the input operation is included in the text input area; an activation means for activating a keyboard when the operation position is included in the text input area; A communication device comprising:

9. 1. A method for controlling an information processing device that provides a rendering result of a web page to a communication device based on a request from the communication device, comprising: an analyzing step of analyzing the web page to determine a text input area; a transmitting step of transmitting the drawing result and information of the text input area to the communication device; 13. A method for controlling an information processing apparatus comprising the steps of:

10. A method for controlling a communication device that displays a rendering result of a web page received from an information processing device, comprising: a receiving step of receiving information on a text input area in the drawing result from the information processing device; a receiving step of receiving an input operation from a user; a determination step of determining whether an operation position of the input operation is included in the text input area; an activation step of activating a keyboard when the operation position is included in the text input area; 13. A method for controlling a communication device, comprising:

11. A program for causing a computer to execute the control method according to claim 9 or 10.

Images