Application remote control system, application execution control device, and application execution control method

Abstract

To provide an application remote control system, an application execution control device, and an application execution control method that can test a native application by using a website test tool.SOLUTION: An application remote control system includes: a terminal image provision unit 12 that provides, to a user terminal 3, a terminal image generated by making a UI operation element as an HTML element related to UI transparent and superimposing it on a background image showing the appearance of an operation target terminal 2 including the UI; and an application operation control unit 11 that operates, when event information of a user operation performed on the UI operation element is acquired from the user terminal 3, a native application 2a on the operation target terminal 2. Then, by performing the user operation on the UI operation element superimposed on the background image via a website test tool, an operation test of the native application 2a installed in the operation target terminal 2 can be executed.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an application remote control system, an application execution control device, and an application execution control method, and is particularly suitable for use in a cloud system that enables remote execution of actual device verification of native applications installed on a target terminal. [Background technology]

[0002] Conventionally, cloud services have been provided that enable remote testing of devices such as smartphones and tablets (see, for example, Non-Patent Document 1). These cloud services enable users to remotely operate applications installed on devices on the cloud from their personal computers or the like, and check the screen and perform operation tests in the same way as on a real device.

[0003] Currently, there are a variety of tools available for testing application programs. They can be broadly divided into testing tools for native applications (hereafter referred to as "apps") installed on personal computers (PCs) or smartphones, and testing tools for websites. App testing tools use hierarchical structure information of the application UI based on user operations on buttons and other displayed input elements to create test code and execute tests. In contrast, website testing tools use hierarchical structure information of the DOM (Document Object Model) of HTML elements on web pages to create test code and execute tests.

[0004] However, in the cloud service mentioned above, the overall appearance of the smartphone, including the display and hard keys, is displayed as an image in the HTML canvas element, and does not contain any hierarchical structure information. As a result, commercially available testing tools are currently unable to directly run tests on the images provided by the cloud service.

[0005] Because testing tools for websites and apps use different hierarchical structure information, the range of targets they support is generally limited. Testing tools are divided into three categories: those that support only websites, those that support only apps, and those that support both. Supported operating systems are also generally limited, meaning users must not only select and use the appropriate tool depending on the content of the test, but also learn how to operate multiple tools. Additionally, with new smartphone models and new OS versions being released daily, users who do not have the appropriate testing tools are unable to run tests or find it time-consuming to set up a test environment.

[0006] There is known a technology that displays a web image transparently on top of a native image, enabling operations on parts of the web image (see, for example, Patent Documents 1 and 2). The information processing device described in Patent Document 1 makes at least a partial area of ​​the native image and the web image transparent or semi-transparent and overlays them, so that element images other than the background, such as icons, UI components, text information, figures, and thumbnail images, which are components of each image, are simultaneously displayed without being hidden, and when a user performs an operation on the web image to update at least a part of the web image, the native image is notified of this.

[0007] In the display device described in Patent Document 2, the web app is displayed so that the area for displaying the native app becomes transparent in the display area of ​​the web app, and the native app is displayed over the transparent area. When a user input is received in the display area of ​​the native app, the display area of ​​the web app does not change, and after the native app is executed, the web app is displayed in the display area of ​​the web app according to the web app. *Patent document 2 refers to it as a "web application," so we have reverted to the original term. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-95256 [Patent Document 2] Japanese Patent Application Laid-Open No. 2018-41240 [Non-patent literature]

[0009] [Non-Patent Document 1] "Cloud service that allows remote testing of actual devices" (Internet search, March 27, 2024) <URL:https: / / appkitbox.com / ) Summary of the Invention [Problem to be solved by the invention]

[0010] The present invention has been made to solve the above problems, and aims to enable testing of native apps using a website testing tool. [Means for solving the problem]

[0011] In order to solve the above-mentioned problems, in the present invention, a terminal image is provided to a user terminal, which is generated by transparently overlaying UI operation elements, which are HTML elements related to the UI, on a background image that represents the appearance of the terminal including the UI of the terminal to be operated. Then, when event information of a user operation performed on the UI operation elements is acquired from the user terminal, a native app installed on the terminal to be operated is run based on the acquired event information, and the terminal image is provided to the user terminal, which is generated by transparently overlaying the UI operation elements on a background image whose appearance reflects the execution result. [Effects of the Invention]

[0012] According to the present invention configured as described above, a native app installed on a target device can be tested by user operations performed on UI operation elements superimposed on a background image, thereby enabling testing of the native app using a website test tool. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an application remote control system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram showing an example of a functional configuration of a server device (application execution control device) according to the present embodiment. FIG. [Figure 3] FIG. 4 is a diagram illustrating an example of a background image generated by a background image generating unit of the present embodiment. [Figure 4] 10A and 10B are diagrams illustrating an example of a terminal image generated by a UI operation element generating unit of the present embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of UI tree information. [Figure 6] FIG. 10 is a diagram illustrating an example of DOM tree information. [Figure 7] 10 is a flowchart illustrating an example of the overall operation of the server device according to the present embodiment. [Figure 8] 10 is a flowchart illustrating an example of the operation of a terminal image providing unit according to the present embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of intermediate tree information. [Figure 10] FIG. 10 is a diagram illustrating another example of the functional configuration of the application remote control system according to the present embodiment. [Figure 11] FIG. 10 is a diagram illustrating another example of the functional configuration of the application remote control system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment of the present invention will be described below with reference to the drawings. Fig. 1 is a diagram showing an example of the overall configuration of an application remote control system according to this embodiment. As shown in Fig. 1, the application remote control system of this embodiment includes a server device 1 (corresponding to an application execution control device), one or more operation target terminals 2 -1 ,2 -2 , ··· (sometimes collectively referred to as operation target terminal 2) and a user terminal 3.

[0015] The server device 1 and the operation target terminal 2 are connected by a wired cable such as USB or wireless communication means such as Bluetooth (registered trademark). The server device 1 and the user terminal 3 are connected by a communication network 5 such as the Internet and / or a mobile phone network.

[0016] The operation target terminal 2 is configured by, for example, a smartphone or a tablet. In this embodiment, a plurality of operation target terminals 2 -1 ,2 -2 , are connected to the server device 1. A plurality of operation target terminals 2 -1 ,2 -2 ,... are different models, and the operating systems installed are diverse, including Android (registered trademark), iOS (registered trademark), BlackBerry OS (BlackBerry is a registered trademark), and Windows Phone (registered trademark).

[0017] The user terminal 3 is a terminal used by a user who performs an operation test on the operation target terminals 2, and is configured, for example, as a personal computer, a smartphone, or a tablet. The user terminal 3 is equipped with a web browser 3a. The user accesses the server device 1 using the web browser 3a and performs an operation test on the operation target terminals 2 through a test execution screen provided by the server device 1. -1 ,2 -2 , . . . and perform an operation test. At this time, the user can install any native application 2a in the operation target terminal 2 and perform an operation test of that native application 2a.

[0018] For example, the native application 2a is actually run on the operation target terminal 2, and the same image as that displayed on the display of the operation target terminal 2 at that time is provided from the server device 1 to the web browser 3a for display. Then, via the image displayed on the web browser 3a, the UI provided on the screen by the native application 2a of the operation target terminal 2 is remotely operated from the user terminal 3, and an image showing the execution result of the native application 2a running as a result is also provided from the server device 1 to the web browser 3a for display.

[0019] In this way, it is possible to remotely verify on the user terminal 3 whether the native application 2a operates normally on the operation target terminal 2. Also, by operating the UI via a website test tool, it is possible to execute an operation test of the native application 2a in accordance with test code that is generated by recording the manual operations.

[0020] Fig. 2 is a block diagram showing an example of the functional configuration of a server device 1 (application execution control device) according to this embodiment. As shown in Fig. 2, the server device 1 of this embodiment has, as its functional configuration, an application operation control unit 11 and a terminal image providing unit 12. These functional blocks 11 to 12 execute the processes described below through cooperation between hardware and software. For example, the processes of the functional blocks 11 to 12 are executed by the operation of a program stored in a storage medium such as RAM, ROM, a hard disk, or a semiconductor memory under the control of a microcomputer configured to include a CPU, RAM, ROM, etc.

[0021] The application operation control unit 11 controls the operation of the native application 2a based on event information related to the operation of the native application 2a transmitted from the user terminal 3. For example, when the application operation control unit 11 acquires event information related to the operation of launching the native application 2a from the user terminal 3, the application operation control unit 11 executes a process of launching the native application 2a based on the acquired event information.

[0022] Furthermore, when the application operation control unit 11 acquires, from the user terminal 3, event information of a user operation performed on a UI operation element (details will be described later) included in a terminal image provided to the web browser 3a by the terminal image providing unit 12, the application operation control unit 11 causes the native application 2a to execute an operation corresponding to the UI operation based on the acquired event information. That is, when the application operation control unit 11 acquires, from the user terminal 3, event information of a user operation detected by the web browser 3a of the user terminal 3, the application operation control unit 11 generates virtual operation instruction information for the operation target terminal 2 based on the acquired event information, and supplies the generated operation instruction information to the operation target terminal 2 to operate the native application 2a.

[0023] The terminal image providing unit 12 provides the web browser 3a of the user terminal 3 with a terminal image generated by overlaying transparent UI operation elements, which are HTML elements related to the UI, on a background image that represents the appearance of the operation target terminal 2, including the UI. The UI included in the operation target terminal 2 is a UI provided by a native application 2a installed on the operation target terminal 2. The background image representing the appearance of the operation target terminal 2 is, for example, an image captured by a camera from a position looking directly at the operation target terminal 2, and may also include the hard keys of the operation target terminal 2. Note that the background image representing the appearance of the operation target terminal 2 may be an image directly acquired from the inside through an interface provided in the operation target terminal 2, or a screenshot image of a screen displayed on the display of the operation target terminal 2.

[0024] When the application operation control unit 11 executes the operation of the native application 2a, the terminal image providing unit 12 provides the user terminal 3 with a terminal image generated by overlaying transparent UI operation elements on a background image whose appearance reflects the execution result. When an image captured by a camera is used as the background image representing the appearance of the operation target terminal 2, when the application operation control unit 11 executes the operation of the native application 2a, the terminal image providing unit 12 is notified of the execution, and the terminal image providing unit 12 controls the camera to capture an image of the appearance of the operation target terminal 2. The user terminal 3 is then provided with a terminal image generated by overlaying transparent UI operation elements on the background image acquired by the image capture. Note that when an internally acquired image or a screenshot image of the operation target terminal 2 is used as the background image representing the appearance of the operation target terminal 2, when the application operation control unit 11 executes the operation of the native application 2a, the terminal image providing unit 12 is notified of the execution, and the terminal image providing unit 12 controls the operation target terminal 2 to acquire an image. The user terminal 3 is then provided with a terminal image generated by overlaying transparent UI operation elements on the acquired background image.

[0025] As shown in FIG. 2, the terminal image providing unit 12 has, as more specific functional configurations, a background image generating unit 12a that generates a background image, a UI operation element generating unit 12b that generates UI operation elements, and a UI operation element superimposing unit 12c that makes the UI operation elements transparent and superimposes them on the background image.

[0026] As described above, the background image generation unit 12a generates a background image by, for example, capturing an image of the target terminal 2 using a camera installed in a position facing the target terminal 2. The generated background image is then sent to the web browser 3a of the user terminal 3 for display. FIG. 3 is a diagram showing an example of a background image generated by the background image generation unit 12a. Here, an example of background image 31 is shown in a state where the native application 2a of the calculator has been launched and the operation screen is displayed.

[0027] The UI operation element generation unit 12b converts the UI tree information of the native application 2a acquired from the operation target terminal 2 into DOM tree information of HTML elements, and generates UI operation elements, which are HTML elements related to the UI, based on the DOM tree information. The UI tree information is a UI library in which multiple UIs owned by the native application 2a are modeled as a tree structure, and is created in a data format unique to the OS of the operation target terminal 2. For example, in the case of Android, the UI tree information is created in XML format. FIG. 5 is a diagram showing an example of the UI tree information. On the other hand, the DOM tree information is a UI library in which multiple UIs owned by the native application 2a are modeled using HTML elements. FIG. 6 is a diagram showing an example of the DOM tree information.

[0028] The UI operation element superimposing unit 12c makes transparent the UI operation elements generated by the UI operation element generating unit 12b from the HTML elements of the DOM tree information, and displays the UI operation elements superimposed on the background image already displayed in the web browser 3a by the background image generating unit 12a. FIG. 4 shows an example of a terminal image generated by superimposing the transparent UI operation elements on the background image. Here, an example is shown in which an HTML div element 41, an input element 42, and a button element 43 are superimposed on a background image 31 in which the calculator's native application 2a has been launched and an operation screen is displayed. In FIG. 4, the UI operation elements are shown with a bold frame for the sake of explanation, but in reality, they are made transparent and are not visible to the user. In other words, the appearance is the same as that of FIG. 3.

[0029] A user of the user terminal 3 can operate individual input elements 42 or button elements 43 via, for example, a website test tool. The website test tool is a tool that can record manual operations on UI operation elements, adjust a test script based on the record, execute a test, and generate a report of the execution results.

[0030] For example, when a user operates one of the button elements 43, the web browser 3a detects the operation and transmits event information of the user operation generated, for example, using JSON-RPC to the server device 1. When the application operation control unit 11 of the server device 1 acquires the event information such as JSON-RPC from the user terminal 3, it converts it into an agent event that conforms to the method of the agent provided in the server device 1. Next, the agent in the server device 1 and the daemon in the operation target terminal 2 cooperate to generate virtual operation instruction information for the operation target terminal 2 based on the agent event, and supplies the generated operation instruction information to the operation target terminal 2 to run the native application 2a.

[0031] After the application operation control unit 11 executes the operation of the native application 2a, the background image generation unit 12a generates a background image whose appearance reflects the execution result, and transmits the generated background image to the web browser 3a of the user terminal 3 for display. The UI operation element generation unit 12b generates UI operation elements in the same manner as described above. The UI operation element superimposition unit 12c then makes the UI operation elements generated by the UI operation element generation unit 12b transparent, and displays the UI operation elements superimposed on the background image already displayed on the web browser 3a by the background image generation unit 12a.

[0032] 7 and 8 are flowcharts showing an example of the operation of the server device 1 according to this embodiment configured as described above. Fig. 7 shows an example of the overall operation of the server device 1, and Fig. 8 shows an example of the operation of the terminal image providing unit 12.

[0033] 7, the application operation control unit 11 determines whether or not event information of an operation to launch the native application 2a has been acquired from the web browser 3a of the user terminal 3 (step S1). If the event information of the application launch operation has been acquired, the application operation control unit 11 executes a process to launch the native application 2a based on the acquired event information (step S2).

[0034] The terminal image providing unit 12 provides the web browser 3a with a terminal image generated by overlaying transparent UI operation elements, which are HTML elements related to the UI, on a background image that represents the appearance of the initial screen (including the UI) displayed upon launch of the native application 2a (step S3). Details of this process are shown in the flowchart of FIG.

[0035] Thereafter, the application operation control unit 11 determines whether or not event information of a user operation performed on a UI operation element has been acquired from the user terminal 3 (step S4). If the event information of a UI operation has not been acquired, the process proceeds to step S7. On the other hand, if the event information of a UI operation has been acquired, the application operation control unit 11 executes a process for causing the native application 2a to perform an operation corresponding to the UI operation based on the acquired event information (step S5).

[0036] When the application operation control unit 11 executes the operation of the native application 2a, the terminal image providing unit 12 provides the web browser 3a of the user terminal 3 with a terminal image generated by overlaying the UI operation elements with transparency on a background image whose appearance reflects the execution result (step S6). The processing performed here is the same as the processing of step S3, the details of which are shown in the flowchart of Fig. 8. Then, the processing proceeds to step S7.

[0037] In step S7, the application operation control unit 11 determines whether or not event information of an operation to terminate the native application 2a has been acquired from the web browser 3a of the user terminal 3. If event information of an operation to terminate the application has not been acquired, the process returns to step S4. On the other hand, if event information of an operation to terminate the application has been acquired, the application operation control unit 11 executes a process to shut down the native application 2a based on the acquired event information (step S8), and ends the process of the flowchart shown in FIG.

[0038] 8, the background image generation unit 12a generates a background image by, for example, controlling a camera to capture an external image of the operation target terminal 2, and transmits the generated background image to the web browser 3a to display it (step S11). Next, the UI operation element generation unit 12b acquires UI tree information of the native application 2a from the operation target terminal 2 (step S12), converts the acquired UI tree information into DOM tree information of HTML elements, and generates UI operation elements based on the converted DOM tree information (step S13). Next, the UI operation element superimposition unit 12c makes the UI operation elements generated by the UI operation element generation unit 12b transparent and displays the UI operation elements superimposed on the background image (step S14).

[0039] As described above in detail, the server device 1 of this embodiment provides the user terminal 3 of the test executor with a terminal image generated by transparently overlaying UI operation elements, which are HTML elements related to the UI, on a background image that represents the appearance of the operation target terminal 2, including the UI. When event information of user operations performed on the UI operation elements is acquired from the user terminal 3, the server device 1 runs the native application 2a installed on the operation target terminal 2 based on the acquired event information, and provides the user terminal 3 with a terminal image generated by transparently overlaying the UI operation elements on a background image whose appearance reflects the execution results.

[0040] According to this embodiment configured as above, by performing user operations on UI operation elements superimposed on a background image via a website test tool, it is possible to execute an operation test of the native application 2a installed on the operation target terminal 2. This makes it possible to test the native application 2a using the website test tool.

[0041] In the above embodiment, an example has been described in which UI tree information, which has a different structure depending on the OS of the operation target terminal 2, is directly converted into DOM tree information of HTML elements. However, the present invention is not limited to this. For example, the UI operation element generation unit 12b may convert the UI tree information of the native application 2a, which is dependent on the OS of the operation target terminal 2, into intermediate tree information independent of the OS of the operation target terminal 2, and then convert the intermediate tree information into DOM tree information of HTML elements. Fig. 9 is a diagram showing an example of intermediate tree information. Here, an example in which the intermediate tree information is configured in JSON format is shown.

[0042] When converting into intermediate tree information, the UI operation element generation unit 12b may optimize the intermediate tree information by combining text information generated by performing character recognition on the background image (a photographed image showing the appearance of the terminal 2 to be operated) generated by the background image generation unit 12a with the intermediate tree information, and convert the optimized intermediate tree information into DOM tree information.

[0043] Depending on the native application 2a being tested, letters and numbers displayed in the UI may be represented as images rather than text information. In this case, information corresponding to the letters and numbers displayed as images may be missing from the intermediate tree information converted from the UI tree information. The optimization process described above refers to a process of performing character recognition processing on an image captured by the target device 2 to fill in the missing letters and numbers in the intermediate tree information converted from the UI tree information with recognized text information.

[0044] Although such optimization processing can be performed on UI tree information, it is necessary to create an algorithm for performing the optimization processing for each OS, because UI tree information has a structure that varies depending on the OS of the target terminal 2. In contrast, if the optimization processing is performed on the intermediate tree information after conversion into intermediate tree information that is independent of the OS, there is an advantage in that it is only necessary to prepare one algorithm for performing the optimization processing.

[0045] The application remote control system of this embodiment is suitable for use in a use case in which the operation target terminal 2 is remotely operated to perform an operation test of the native application 2a, but it can also be applied to use cases other than testing.

[0046] Furthermore, in the above embodiment, a configuration example has been described in which the server device 1 has all the functions of the terminal image providing unit 12 and the application operation control unit 11, but a configuration in which some or all of these functions are provided in the user terminal 3 may also be adopted. For example, as shown in Fig. 10, a configuration in which the web browser 3a of the user terminal 3 has the function of the UI operation element superimposing unit 12c may also be adopted.

[0047] 11, a configuration may be adopted in which all of the functions of the terminal image providing unit 12 and the application operation control unit 11 are provided in the web browser 3a of the user terminal 3. In this case, the server device 1 is omitted, and the user terminal 3 and the operation target terminal 2 are connected via a communication network 5. For example, by providing the web browser 3a with an API (Application Programming Interface) that directly operates a USB-compatible device, it is possible to remotely operate the operation target terminal 2 directly from the web browser 3a, and it is also possible to implement all of the functions of the terminal image providing unit 12 and the application operation control unit 11 in the web browser 3a.

[0048] Furthermore, the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereby. In other words, the present invention can be carried out in various forms without departing from the gist or main characteristics thereof. [Explanation of symbols]

[0049] 1. Server device (application execution control device) 2. Target device 2a Native App 3. User terminal (application execution control device) 3a Web browser 11 App operation control unit 12. Terminal Image Providing Department 12a Background image generation section 12b UI operation element generation section 12c UI operation element overlay

Claims

1. a terminal image providing unit that provides a user terminal with a terminal image generated by making UI operation elements, which are HTML elements related to the UI, transparent and overlaying them on a background image that represents the appearance of the target terminal including the UI; an application operation control unit that acquires, from the user terminal, event information of a user operation performed on the UI operation element, and operates a native application installed on the operation target terminal based on the acquired event information; When the application operation control unit executes the operation of the native application, the terminal image providing unit provides the user terminal with a terminal image generated by making the UI operation elements transparent and superimposing them on a background image whose appearance reflects a result of the execution. An app remote control system characterized by:

2. 2. The application remote control system according to claim 1, wherein the application operation control unit acquires, from the user terminal, event information of the user operation detected by a web browser of the user terminal, generates virtual operation instruction information for the operation target terminal based on the acquired event information, and supplies the generated operation instruction information to the operation target terminal to operate the native application.

3. The terminal image providing unit a background image generation unit that generates the background image; a UI operation element generation unit that generates the UI operation element; a UI operation element superimposing unit that makes the UI operation element transparent and superimposes it on the background image, The UI operation element generation unit converts UI tree information of the native application into DOM tree information of the HTML elements, and generates the UI operation elements based on the DOM tree information.

3. The application remote control system according to claim 1 or 2.

4. The terminal image providing unit a background image generation unit that generates the background image; a UI operation element generation unit that generates the UI operation element; a UI operation element superimposing unit that makes the UI operation element transparent and superimposes it on the background image, The UI operation element generation unit converts UI tree information of the native application into intermediate tree information that is independent of the OS of the target terminal, then converts the intermediate tree information into DOM tree information of the HTML elements, and generates the UI operation element based on the DOM tree information.

3. The application remote control system according to claim 1 or 2.

5. The application remote control system according to claim 4, wherein the UI operation element generation unit optimizes the intermediate tree information by synthesizing text information generated by character recognition of the background image generated by the background image generation unit with the intermediate tree information, and converts the optimized intermediate tree information into DOM tree information of the HTML elements.

6. a terminal image providing unit that provides a user terminal with a terminal image generated by making UI operation elements, which are HTML elements related to the UI, transparent and overlaying them on a background image that represents the appearance of the target terminal including the UI; an application operation control unit that acquires, from the user terminal, event information of a user operation performed on the UI operation element, and operates a native application installed on the operation target terminal based on the acquired event information; When the application operation control unit executes the operation of the native application, the terminal image providing unit provides the user terminal with a terminal image generated by making the UI operation elements transparent and superimposing them on a background image whose appearance reflects a result of the execution. An application execution control device characterized by:

7. a first step in which a terminal image providing unit of the application execution control device provides a user terminal with a terminal image generated by overlaying a background image showing an appearance of the target terminal including a UI, with UI operation elements, which are HTML elements related to the UI, in a transparent manner; a second step in which an application operation control unit of the application execution control device acquires, from the user terminal, event information of a user operation performed on the UI operation element, and operates a native application installed on the operation target terminal based on the acquired event information; a third step of the terminal image providing unit providing, to the user terminal, a terminal image generated by making the UI operation elements transparent and superimposing them on a background image whose appearance reflects the execution result of the native application.

10. An application execution control method comprising:

Images