Automatic interaction method and electronic equipment

Abstract

The embodiment of the invention provides an automatic interaction method and electronic equipment, the method is executed by the electronic equipment, and the method comprises the following steps: receiving a first operation input by a user, the first operation being used for triggering the electronic equipment to automatically execute a first service; under the condition that a first application program corresponding to the first service exists in a preset white list, the starting thread polls an application program main window created by the electronic equipment in a first preset duration, and the application program main window comprises a corresponding interaction window when the first service is executed; if the interactive window is polled and the window coordinates of the interactive window are located on the main screen of the electronic equipment, the interactive window is moved into a smart window to be displayed, and the smart window is located in a partial area of the main screen of the electronic equipment and used for displaying the interactive process that the electronic equipment automatically executes the first service. Therefore, the interaction window is verified and judged in a thread polling mode, an abnormal scene that the interaction window does not move to the intelligent window is avoided, and the stability of an automatic interaction function is improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an automated interaction method and electronic equipment. Background Art

[0002] Currently, users can use electronic devices (such as personal computers (PCs)) to perform a variety of operations, such as installing or uninstalling applications, searching browser content, and writing documents. These operations usually require the user's full participation, which imposes a heavy workload on the user.

[0003] Therefore, there is an urgent need for an automated interaction method that can automatically interact with electronic devices and perform operations related to user instructions without the need for full user participation, thereby improving the user experience. Summary of the Invention

[0004] The present application provides an automated interaction method and electronic device, which can improve the stability of automated interaction functions.

[0005] In the first aspect, the present application provides an automated interaction method, which is executed by an electronic device, including: receiving a first operation input by a user, the first operation being used to trigger the electronic device to automatically execute a first business; when a first application corresponding to the first business exists in a preset whitelist, starting a thread to patrol the application main window created by the electronic device within a first preset time period, the application main window including an interactive window corresponding to when executing the first business; if the interactive window is patrolled and the window coordinates of the interactive window are located on the main screen of the electronic device, the interactive window is moved to a smart window for display, and the smart window is located in a partial area of ​​the main screen of the electronic device, for displaying the interactive process of the electronic device automatically executing the first business.

[0006] In this implementation, in order to reduce the amount of repetitive work that users do when operating electronic devices, the electronic device can be made to automatically execute the interaction process of related services. Then, when the electronic device receives an operation from the user to trigger the electronic device to automatically execute the first service, if the application corresponding to the first service exists in the preset whitelist, a thread can be started to patrol the main application window currently created by the electronic device. In some implementations, the identifier of the application can be saved in the preset whitelist, that is, when the application in the whitelist is automatically interacted with, a thread can be started to patrol. When the interactive window corresponding to the first service is patrolled and the window coordinates of the interactive window are located on the main screen, the electronic device can automatically move the interactive window to the smart window for display. In this way, the interactive window is no longer displayed on the main screen of the electronic device, and the user can perform other operations on the main screen without being affected by the display of the smart window. In this way, by verifying and judging the interactive window in a thread patrol manner, it is possible to avoid abnormal scenarios where the interactive window is not moved to the smart window but is displayed on the main screen, thereby improving the stability of the automated interactive function.

[0007] In some implementations, the first preset duration can be set as a general duration from the creation to the display of an application window. The first preset duration can be measured by a timer. If the first preset duration is not reached, the electronic device can continue to patrol the main window of the application through a thread. If the first preset duration has been reached, the electronic device can end the above process.

[0008] In some implementations, the first service is installing or uninstalling a first application, that is, the electronic device can automatically execute the installation or uninstallation process of the application according to a triggering operation of the user.

[0009] In combination with the first aspect, in some implementations of the first aspect, the above-mentioned moving the interactive window to the smart window for display includes: changing the window coordinates of the interactive window to corresponding coordinates on the virtual screen, wherein the virtual screen is a virtual screen created by the electronic device when receiving the first operation; displaying the interactive window on the virtual screen; taking a screenshot of the display interface of the virtual screen, and mapping it to the smart window for display frame by frame.

[0010] Generally, when creating an interactive window corresponding to a first service, the electronic device can determine the corresponding window size, window coordinates, window identifier (e.g., handle), and other information. By default, interactive windows created by the electronic device are displayed on the main screen. If the interactive window is to be moved to a smart window for display, the window coordinates must be modified. In this implementation, after receiving the first operation, the electronic device can also create a virtual screen as the virtual screen for automated interaction. The smart window can be a projection of the virtual screen. Specifically, the display interface on the virtual screen can be captured and sent frame by frame to a designated location on the main screen of the electronic device for display, thereby displaying the smart window. After creating the interactive window, the interactive window can be moved to the virtual screen for display. This allows screenshots of the display interface on the virtual screen to capture the interface corresponding to the interactive window. When the smart window is projected for display, the interactive window can also be displayed. Moving the interactive window to the virtual screen for display can be accomplished by the electronic device modifying the window coordinates of the interactive window to the corresponding coordinates on the virtual screen. In this way, the electronic device can display the interactive window within the smart window, dynamically displaying the automated interaction process.

[0011] In combination with the first aspect, in some implementations of the first aspect, the size of the virtual screen is the same as the size of the display screen of the electronic device.

[0012] Among them, the size of the above-mentioned virtual screen can be the same as the size of the display screen of the electronic device, or it can be different from the size of the electronic device, as long as the mapping relationship between the virtual screen and the smart window can be determined.

[0013] In combination with the first aspect, in some implementations of the first aspect, when the interactive window is patrolled and the window coordinates of the interactive window are located on the main screen of the electronic device, the method further includes: hiding the interactive window on the main screen.

[0014] Considering that in some abnormal scenarios, when the electronic device changes the window coordinates of the interactive window, the interactive window may appear to flash on the main screen, affecting the user experience. The electronic device can first hide the interactive window on the main screen and then change the window coordinates of the interactive window. This can avoid the interactive window flashing on the main screen.

[0015] In combination with the first aspect, in some implementations of the first aspect, after the first operation of receiving user input, the method further includes: starting a main process to execute the first business; suspending the main process when starting to create the interactive window; and unsuspending the main process after starting the thread to patrol the main window of the application.

[0016] Among them, after receiving the above-mentioned first operation input by the user, the electronic device can start the main process of executing the first business, which is used to execute the first business in the background. Since this application requires starting a thread to patrol the main window of the application to move the interactive window to the smart window display, when starting to create the interactive window (that is, before starting the thread to patrol the main window of the application), the main thread started is suspended first, the corresponding application is suspended, and the application corresponding to the interactive window is judged to be in the preset white list. This can prevent the main thread from creating and displaying the interactive window on the main screen too quickly. If the application corresponding to the interactive window is in the preset white list, the electronic device can start a thread to patrol the created main window of the application. At this time, the main thread can also be unsuspended to start the main process to process the interactive window. In this way, within the first preset time period thereafter, on the one hand, the main process processes the interactive window, and on the other hand, the thread patrols the main window of the application of the electronic device. If the thread patrols the interactive window, its window coordinates can be judged and changed to ensure that the window coordinates of the interactive window can eventually be changed to the virtual screen.

[0017] In combination with the first aspect, in some implementations of the first aspect, the above method also includes: if the interactive window is not patrolled, or if the interactive window is patrolled but the window coordinates of the interactive window are not located on the main screen of the electronic device, waiting for a second preset time length; after the second preset time length, continuing to patrol the application main window.

[0018] Since the electronic device can create multiple application windows, these multiple windows are patrolled in sequence, so the above-mentioned interactive window may be patrolled at a certain moment. In this case, if the interactive window is not patrolled, you can wait for the second preset time length first, or, if the interactive window is patrolled but the window coordinates of the interactive window are not located on the main screen, that is, it has been changed to the virtual screen, you can also wait for the second preset time length. Then, after the second preset time length, continue to patrol the main window of the application until the first preset time length is reached. Continuously patrolling the main window of the application can avoid the phenomenon of the interactive window being reset back to display on the main screen.

[0019] In combination with the first aspect, in some implementations of the first aspect, the above method also includes: intercepting the created interactive window when the first application does not exist in the preset whitelist; changing the window coordinates of the interactive window to corresponding coordinates on the virtual screen; displaying the interactive window on the virtual screen; taking a screenshot of the display interface of the virtual screen, and mapping it to the smart window frame by frame for display.

[0020] The above implementation method describes the scenario where the first application exists in a preset whitelist. The interactive window of the first application can be moved to the smart window for display by using a thread patrol of the main window of the application. If the first application does not exist in the preset whitelist, the electronic device can move the interactive window of the first application to the smart window for display by intercepting the interactive window. When intercepting the interactive window, the identifier of the interactive window (such as a handle) can be obtained, and then the window coordinates of the interactive window can be changed to the coordinates corresponding to the virtual screen. When taking a screenshot of the display interface of the virtual screen, the content of the interactive window can be captured accordingly, and then mapped to the smart window for display frame by frame.

[0021] In combination with the first aspect, in some implementations of the first aspect, intercepting the created interactive window includes: intercepting the interactive window using a Hook interception technology.

[0022] Among them, when intercepting the interactive window, the Hook interception technology can be used to execute it. The Hook interception technology can intercept the execution function related to the interactive window and obtain the handle and other information of the interactive window. After intercepting the interactive window, its window coordinates can be modified.

[0023] In combination with the first aspect, in some implementations of the first aspect, the preset whitelist stores identifiers of applications whose windows cannot be intercepted by the Hook interception technology.

[0024] As can be seen from the above implementation, if the first application is on the preset whitelist, the interactive window is processed using a thread patrol method. If the first application is not on the preset whitelist, the interactive window is processed using the hook interception technology. Therefore, the application corresponding to the window that cannot be intercepted by the hook can be stored in the whitelist. The complementary execution of these two methods can further improve the stability of the electronic device's automated interactive function.

[0025] In combination with the first aspect, in some implementations of the first aspect, after moving the interactive window to be displayed in the smart window, the method further includes: simulating a user's click operation on the interactive window to automatically execute the first business.

[0026] Since the process of executing the first business (such as the installation process of the first application) usually involves confirmation operations on some options, in order to improve visibility, the electronic device can also display indicator sources such as a cursor or arrow on the interactive window of the smart window, that is, simulate the user's click operation, thereby clearly displaying the automated operation process of the electronic device and improving visibility.

[0027] In combination with the first aspect, in some implementations of the first aspect, before receiving the first operation input by the user, the method also includes: displaying a first interface, on which an instruction input control is presented; in response to the instruction input by the user through the instruction input control, displaying an operation control corresponding to triggering the execution of the first business; and receiving the first operation through the operation control.

[0028] In addition to the above-mentioned automated interaction process, the electronic device can also display a first interface for the user to input the above-mentioned first operation to trigger the electronic device to automatically execute the first business. Among them, the first interface can be displayed by a UI intelligent body application, and the UI intelligent body can receive instructions input by the user, that is, an instruction input control can be presented on the first interface, such as presenting the instruction input control in the form of a dialog box. After the user enters the instruction through the instruction input control, the UI intelligent body can respond to the user's instruction and perform related searches to answer the instruction entered by the user, such as displaying the response information in the form of a card in the dialog box. In this implementation, the display position of the response information can also display the operation control corresponding to the trigger execution of the first business. For example, for the business of installing the first application, a download control can be displayed. If the user clicks on the operation control, the electronic device is triggered to automatically execute the first business. In this way, through visual interface operations, greater convenience can be provided for user operations and the user experience can be improved.

[0029] In a second aspect, the present application provides a device, which is included in an electronic device and has the function of implementing the electronic device behavior described in the first aspect and possible implementations of the first aspect. The function can be implemented through hardware or through hardware executing corresponding software implementations. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, etc.

[0030] In a third aspect, the present application provides an electronic device, the electronic device comprising: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute any one of the methods in the technical solution of the first aspect.

[0031] In a fourth aspect, the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute the method in the first aspect and any possible implementation thereof.

[0032] Optionally, the chip system also includes a memory, and the memory is connected to the processor via circuits or wires.

[0033] Further optionally, the chip system also includes a communication interface.

[0034] In a fifth aspect, the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes any one of the methods in the technical solution of the first aspect.

[0035] In a sixth aspect, the present application provides a computer program product, which includes: a computer program code, which, when the computer program code runs on an electronic device, enables the electronic device to execute any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of a window scenario of a UI Agent running according to an embodiment of the present application; Figure 2 This is a schematic diagram of a scenario of user interaction instructions with a UI Agent provided in an embodiment of the present application; Figure 3 This is a schematic diagram of a scenario in which a UI Agent starts a smart window, provided in an embodiment of the present application; Figure 4 This is a schematic diagram of a scenario in which a UI Agent interacts with an interactive window, provided in an embodiment of the present application; Figure 5 This is a flow chart of an automated interaction method provided in an embodiment of the present application; Figure 6 This is a schematic diagram of the mapping relationship between the main screen of an electronic device and the virtual screen provided in an embodiment of the present application; Figure 7 1 is a schematic diagram of a software module of an electronic device provided in an embodiment of the present application; Figure 8 This is a flowchart of another automated interaction method provided in an embodiment of the present application; Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0038] In the following, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Therefore, a feature specified as "first," "second," or "third" may explicitly or implicitly include one or more of the features.

[0039] Currently, users can use electronic devices (such as PCs) to perform a variety of operations, such as installing or uninstalling applications (APPs) on electronic devices, searching for browser content, and writing documents. When performing these operations, the user is usually required to participate in the entire process, which brings a large workload to the user. For example, if a user wants to install an APP on an electronic device, they need to first download the APP installation package from the download website, and then trigger the APP installation process. During this process, they also need to perform tasks such as selecting the installation path for the APP. Obviously, the user's workload is relatively large, and they will also spend a lot of time on these repetitive tasks.

[0040] Therefore, there is an urgent need for an automated interaction method that can automatically interact with electronic devices and perform operations related to user instructions without the user's full participation, thereby improving the user experience. For example, if a user enters the command "Install APP1", the electronic device will automatically download the installation package of APP1 and automatically execute the installation process. The user can perform other operations during this period, thereby improving the utilization rate of the electronic device and the user experience.

[0041] The automated interaction method provided in the embodiments of the present application can be applied to electronic devices that require interaction with users, such as PCs, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0042] The following first introduces the application scenario of the automated interaction method provided by the embodiment of the present application. Taking the electronic device as a PC as an example, a UI agent (user interface agent, UI Agent) is installed on the PC. The UI Agent can receive instructions input by the user, such as text instructions, voice instructions, etc., and then the UI Agent can automatically perform related operations based on the user instructions. Furthermore, in order to improve the degree of visualization, when starting the automated operation, the electronic device can also display a smart window on the display screen, and the business operations performed by the UI Agent are displayed in the smart window. At the same time, the user can perform other operations on the main screen of the electronic device without being affected by the display of the smart window.

[0043] For example, the UI Agent can be presented in the form of an application, and the application name can also be called PC Assistant, etc., which is not limited in this embodiment of the present application. Figure 1 As shown, when the UI Agent is running, a view window is displayed, which includes functional controls. For example, the functional controls are displayed on the left side of the window. The functional controls may include a new dialog control, a smart search control, a writing assistant control, a reading assistant control, and a smart control control. When a user clicks the new dialog control 11, a command input control 12 may be displayed in the view window, through which the user can enter commands.

[0044] For example, Figure 2 As shown, the instruction input control 12 may include a text input box 121, through which the user can input text instructions. The instruction input control 12 may also include a voice input control 122, and the user can input a voice instruction after clicking the voice input control 122. Taking the user inputting a text instruction as an example, after entering the text instruction, clicking the send button can display the text instruction in the newly created dialog box 21, and the UI Agent can then receive the instruction input by the user. After receiving the above instruction, the UI Agent can perform a related search to respond to the instruction input by the user. For example Figure 2 As shown in FIG, a user enters the command "Install APP1" and sends it. Dialog box 21 also displays the corresponding dialog content of "Install APP1". After the UI Agent searches for the installation package of APP1, it can display the relevant information of the installation package in the dialog box and display the download control 22. If the user clicks the download control 22, the UI Agent will automatically download and install APP1. In some implementations, the UI Agent can display the searched content (such as the relevant information of the installation package) in the form of a card in the dialog box.

[0045] As another example, after the UI Agent receives the instruction input by the user, it can directly respond to the instruction and automatically execute the business operation corresponding to the instruction.

[0046] See also Figure 3 , UI Agent automatically downloads the installation package of APP1 and can display the download progress first (for example Figure 3 The download progress is shown in the card), and the installation process is automatically triggered after the download is completed. At this time, the electronic device can display a smart window on the display screen, and the specific installation process of APP1 is displayed in the smart window. It can be understood that the content displayed in the smart window is not displayed on the main screen of the electronic device, so that the user can perform other operations on the main screen. At the same time, the prompt message "The application will be automatically installed soon. You can view the progress in the floating window without affecting other operations" can also be displayed in the dialog box 21. Among them, the smart window can be displayed in a partial area of ​​the display screen, such as the upper right corner or the upper left corner, etc., and the embodiment of the present application does not limit this. The installation process of APP1 can be dynamically displayed in the smart window, such as displaying the real-time installation progress of APP1, displaying the installation path of APP1, etc. In some implementations, the installation path of APP1 selected by the UI Agent may be the default installation path. In other implementations, the installation path of APP1 selected by the UI Agent may be a path determined based on user habits.

[0047] Since the installation process of APP1 usually involves confirmation operations of some options, such as confirming the installation path, confirming some configuration items, confirming the next operation, etc., these confirmation operations need to be performed manually by the user in the related art, but the embodiment of the present application can be automated by the UI Agent. In order to facilitate the user to view the operation process of the UI Agent, the cursor or arrow and other indicator sources can also be displayed in the smart window, that is, simulating the user's click operation. For example, Figure 4 As shown, the smart window displays the content of confirming the installation path. If the "OK" option is selected, a cursor 41 can be displayed on the option control, so that the automated operation process of the UI Agent can be clearly displayed with better visibility.

[0048] In some implementations, the user can also operate the smart window, such as dragging the smart window to change its display position, clicking the minimize control to make the smart window exit the display on the screen, clicking the close control to close the smart window, etc. Figure 4 In the case of minimizing the control 42, the electronic device can be triggered to hide the smart window, and the electronic device can automatically perform the interactive operation in the background. Figure 4When the smart window is closed by the close control 43, it can be determined that the automated operation process of the UI Agent is closed, that is, the subsequent interactive operations are no longer performed by the UI Agent, and are switched to manual operation by the user. At this time, the interface displayed in the smart window (such as the interface of the APP1 installation process) can be switched to the main screen of the electronic device for the user to manually perform subsequent operations.

[0049] I understand. Figure 1-Figure 4 The diagram shows a scenario of installing an APP using the UI Agent automation function. In addition to this scenario, it can also be applied to scenarios such as using the UI Agent for automated searches (such as browser searches), automated implementation of APP-related functions (such as opening a video APP to play videos, opening a file to extract text content, opening a music APP to play music, etc.). The embodiments of this application do not limit this.

[0050] Based on the above application scenarios, such as Figure 5 As shown, the automated interaction method performed by the electronic device may include the following processes: S11, receiving a first operation input by a user.

[0051] The first operation is an operation that triggers the start of the UI Agent automation process, such as the user's click operation on the download control 22 .

[0052] S12, start the main process.

[0053] The main process here may be related to the business corresponding to the automation process. For example, if the business is to automatically install an APP, the main process may be the process of installing the APP.

[0054] S13, creating an interactive window.

[0055] S14, processing interaction window.

[0056] S15, displaying an interactive window in the smart window.

[0057] As can be seen from the above application scenario, a smart window will be displayed during the automated process. The smart window displays the UI Agent's interactive process, which is also displayed in the form of a window. Therefore, it is necessary to create and process the corresponding interactive window to ultimately display it. For example, during the automatic installation of App1, the smart window displays the interactive window corresponding to the installation process.

[0058] When creating an interactive window, the corresponding window size, window coordinates, window identifier (such as a handle), and other information can be determined. Generally, interactive windows created by electronic devices are displayed on the main screen by default. However, the embodiment of the present application is to display the interactive window in the smart window, and not on the main screen. Therefore, it is necessary to move the interactive window to the smart window for display.

[0059] In some implementations, when the UI Agent automation process is triggered, the electronic device can create a virtual screen as the screen for the UI Agent automation interaction. The size of the virtual screen can be the same as or different from the size of the electronic device's display screen. Then, the smart window can be a projection of the virtual screen, that is, the content displayed on the smart window is the content displayed on the virtual screen, see Figure 6 , shows the mapping relationship between the main screen of the electronic device and the virtual screen. The main screen of the electronic device is the screen where the user actually interacts, and the virtual screen is a screen that is invisible to the user and is used for UIAgent interaction. During the display process, you can take a screenshot of the display interface on the virtual screen, and send it frame by frame to the designated position of the main screen of the electronic device for display, so that the above-mentioned smart window can be displayed. Then, after the interactive window is created as mentioned above, you can move the interactive window to the virtual screen for display. In this way, when you take a screenshot of the display interface on the virtual screen, you can capture the interface corresponding to the interactive window, and then when you project and display the smart window, the interactive window can also be displayed accordingly. Therefore, we have to solve the problem of how to move the interactive window to the virtual screen for display.

[0060] It is understandable that after the electronic device creates a virtual screen, the user can view relevant information of the virtual screen, such as the size of the virtual screen, through the property setting path of the display screen on the electronic device.

[0061] In some implementations, the electronic device can use Hook interception technology to intercept the created interactive window and move it to the virtual screen before the window is displayed on the main screen. Among them, the Hook interception technology can intercept the execution function related to the interactive window, obtain the identifier of the interactive window (such as the handle), and then change the window coordinates of the interactive window to the coordinates corresponding to the virtual screen. That is to say, when creating an interactive window, the default window coordinates are the coordinates corresponding to the main screen of the electronic device. After changing it to the coordinates corresponding to the virtual screen, it will not be displayed on the main screen, but on the virtual screen. In this way, the interactive window can be displayed in the smart window later.

[0062] Exemplarily, the coordinates corresponding to the main screen of the electronic device may be in the range of [0, 0] to [1920, 1080], and the coordinates corresponding to the virtual screen may be set outside the range.

[0063] S16, executing the instruction corresponding to the first operation.

[0064] It is understood that after the interactive window is displayed, the electronic device needs to start executing the instructions corresponding to the above-mentioned first operation and display the execution process through the interactive window. For example, for the above-mentioned example of automatically installing an APP, the electronic device can execute the corresponding installation instructions. In some implementations, if other interactive operations are involved when executing the instructions corresponding to the first operation, such as simulating the confirmation operation of some options with a keyboard and mouse, a corresponding sub-process response operation can be created.

[0065] In some implementations, after the UI Agent completes the automated operation, such as the installation of APP1, the smart window can be automatically closed and the created virtual screen can be destroyed.

[0066] Through the above process, electronic devices can achieve automated interaction, thereby improving the utilization rate of electronic devices and the user experience.

[0067] To implement the above-mentioned automatic interaction method, the electronic device may have the following features: Figure 7 The software module shown. Taking the scenario of installing or uninstalling APP as an example, UI Agent can be an application in the application layer, which can be used as the above Figure 1 The view window shown is displayed. When the UI Agent receives the first operation input by the user, the request instruction can be sent to the lower-level window management module. The window management module may include a high-performance remote procedure call protocol service (GRPC Server). The GRPC Server can be used to respond to the request instructions sent by the UI Agent, such as including installation or uninstallation of APP response, keyboard and mouse response, APP screenshot response and APP exception response.

[0068] In addition, the window management module may also include a window display management module, a virtual screen control module, an APP installation or uninstallation module, a keyboard and mouse control module, an image capture control module, and a window position management module.

[0069] Among them, the window display management module can be used to create and manage smart windows, the virtual screen control module can be used to create and manage virtual screens, the APP installation or uninstallation module can be used to install or uninstall APPs, the keyboard and mouse control module can be used to control the keyboard and mouse operations simulated by the keyboard and mouse operation module on the virtual screen and the operations performed by the user through the physical keyboard and mouse, the image capture control module can be used to control the image capture module to capture the display interface on the virtual screen, such as virtual screen capture, APP window capture, etc., and send it frame by frame to the smart window display, and the window position management module can be used to change the window coordinates of the interactive window to the coordinates corresponding to the virtual screen. The electronic device may also include a Hook management module, which can be used to intercept the interactive window of the application created by the electronic device and move the interactive window to the virtual screen display.

[0070] The workflow of each of the above software modules is as follows: After the window display management module creates the smart window and the virtual screen control module creates the virtual screen, during the installation of the APP by the APP installation or uninstallation module, the window position management module can intercept the interactive window of the APP installation process through the Hook management module, transform the coordinates of the interactive window, and move the interactive window to the virtual screen for display. The image capture control module then controls the screenshot of the display interface on the virtual screen and sends it frame by frame to the smart window for display. While the interactive window is displayed on the virtual screen, the keyboard and mouse control module can also control simulated keyboard and mouse operations, such as simulating clicks to confirm option controls.

[0071] It is understandable that the electronic device may further include a virtual screen driver for servicing the operation of the virtual screen, such as for virtual screen creation or destruction, virtual screen information management, virtual screen mode control, driver loading or unloading, etc.

[0072] In some implementations, the Hook management module may also include an injector and a Hook module. The injector may be used to manage the Hook, such as injection control, management of inter-process communication (IPC), management of Hook installation or uninstallation, etc. The Hook module may be used to perform interactive window identification, interactive window position control, IPC communication, etc.

[0073] exist Figure 7 Based on the software modules shown above, the above Figure 5The process of the automated interaction method shown, however, considering the diversity of applications installed on electronic devices (such as PCs), the window processing mechanisms of applications may be different. The above-mentioned Hook interception technology may not be able to intercept the interactive windows of all applications, resulting in the UI Agent being unable to move the interactive window to the virtual screen display in some scenarios, and it is still displayed on the main screen of the electronic device. Alternatively, in the process from creation to display of the interactive window, processing processes such as rendering are involved. It is possible that after the window coordinates of the interactive window are changed to the coordinates corresponding to the virtual screen, they are reset to the coordinates corresponding to the default main screen during the processing stage, and are still displayed on the main screen of the electronic device. If such an abnormal situation exists, the electronic device will not be able to display the interactive window in the smart window, and the stability of the automated interaction function is weak, which brings certain troubles to user operations.

[0074] In view of this, the automated interaction method provided in the embodiment of the present application can also improve the stability of the UI Agent automated interaction function, so that the interactive window can be stably displayed in the smart window. Figure 8 , which shows a flow chart of another example of an automated interaction method provided in an embodiment of the present application, which may specifically include the following steps: S21, receiving a first operation input by the user and starting the main process.

[0075] The first operation is an operation that triggers the startup of the UI Agent automation process, that is, an operation that triggers the automatic execution of the first service of the application, such as the above-mentioned operation of executing the installation of the application.

[0076] S22, when starting to create the interactive window, suspend the main process.

[0077] S23, determining whether the application corresponding to the interactive window is in a preset whitelist, if so, executing S24, if not, unsuspending the main process.

[0078] Among them, when the electronic device starts to create an interactive window, it first suspends the main thread that is started, suspends the startup of the corresponding application, for example, suspends the installation process of the application, and then determines whether the application corresponding to the interactive window is in the preset white list. This can prevent the main process from being executed too quickly and displaying the interactive window on the main screen. If it is not in the white list, the main process is unsuspended and the subsequent processing and display of the interactive window continues, that is, the process of using Hook interception technology to intercept the created interactive window in S13-15 of the above embodiment is normally executed, and the window is moved to the virtual screen before it is displayed on the main screen. If the application corresponding to the interactive window is in the preset white list, it needs to be subjected to subsequent special processing.

[0079] In some implementations, the preset whitelist may include the identifiers of applications that cannot be intercepted through the Hook interception technology. That is, applications that cannot display interactive windows in the smart window in the above-mentioned abnormal scenarios can be saved in the whitelist and processed separately. Thus, by complementing the execution of the two methods, the stability of the automated interactive function of the electronic device can be further improved. In other implementations, the identifiers of applications that can be intercepted through the Hook interception technology can also be saved in the preset whitelist to avoid the phenomenon of unsuccessful interception.

[0080] S24, starting a thread to patrol the main window of the application created by the electronic device.

[0081] In this embodiment, to avoid abnormal situations such as the window coordinates of the interactive window not being changed to the virtual screen and / or being reset to the coordinates corresponding to the default main screen after being changed, the electronic device can start a thread to patrol the created application main windows. If the patrol reaches the interactive window, its window coordinates are changed. It should be understood that the application main windows created by the electronic device may not be limited to the interactive window, but may also include the main windows of other applications, so these main windows are traversed in a patrol manner.

[0082] S25, unsuspend the main process.

[0083] Since the main process has been suspended, after the thread is started to patrol the main window of the application, the main thread can be unsuspended to start the main process to process the interactive window of the application.

[0084] S26, start the timer and set the timer to the first preset time.

[0085] That is to say, the electronic device is timed to a preset duration, during which the main process processes the interactive window on the one hand, and the thread patrols the main window of the application program on the electronic device on the other hand. Therefore, during the process of the main process processing the interactive window, if the thread patrols the interactive window, its window coordinates can be judged and changed to ensure that the window coordinates of the interactive window can eventually be changed to the virtual screen.

[0086] In some implementations, the preset duration can be set to a general duration from creation to display of an application window, for example, 2 seconds, that is, the electronic device can basically complete the process of window creation and display within 2 seconds.

[0087] S27, determine whether the timer time has expired, if not, execute S28, if the time has expired, end the process.

[0088] That is, if the timer has not expired, the electronic device can continue the patrol process of the application's main window, and when the patrol reaches the interactive window, the window coordinates of the interactive window will be changed to the virtual screen. If the timer has expired, it means that the interactive window has most likely been processed, and the electronic device can destroy the patrol thread, ending the process.

[0089] S28, determine whether the interactive window is patrolled, if not, execute S29, if so, execute S30.

[0090] S29, waiting for a second preset time period, and executing S27.

[0091] When multiple application windows are created on an electronic device, these multiple windows are patrolled in sequence, so the interactive window may be patrolled at a certain moment. In this case, if the interactive window is not patrolled, the second preset time period can be waited for, and then it can be determined whether the interactive window is patrolled. In this way, the created application main window can be continuously polled within the first preset time period.

[0092] In some implementations, the second preset duration may be 10 milliseconds (ms).

[0093] S30, determining whether the window coordinates of the interactive window are located on the main screen, if so, executing S31, if not, executing S29.

[0094] S31, hide the interactive window on the main screen.

[0095] S32, changing the window coordinates of the interactive window to corresponding coordinates on the virtual screen.

[0096] S33, displaying the interactive window on the virtual screen, and executing S27.

[0097] Among them, if the electronic device patrols the interactive window and finds that the window coordinates of the interactive window are located on the main screen, the interactive window can be hidden on the main screen first, such as setting the display properties on the main screen to invisible, so as to avoid the interactive window flashing on the main screen. Then the window coordinates of the interactive window are changed to the corresponding coordinates on the virtual screen, that is, moved to the virtual screen for display. After that, the above S27 is executed in a loop, and the main window of the application is continuously patrolled within the time limit. If the window coordinates of the interactive window are reset to the corresponding coordinates on the main screen, it can be changed again to the corresponding coordinates on the virtual screen to ensure that the interactive window is finally successfully displayed on the virtual screen.

[0098] If the electronic device patrols to the interactive window and finds that the window coordinates of the interactive window are not located on the main screen, that is, it has been changed to the virtual screen, the above S29 process of waiting for the second preset time length can be executed, and then it can be determined whether the interactive window is patrolled.

[0099] It can be understood that after the interactive window is displayed on the virtual screen, the electronic device can take a screenshot of the display interface on the virtual screen and map it frame by frame to the smart window for display.

[0100] It can also be understood that the process of the above embodiment can be executed by the window position management module, that is, when the window position management module determines that the application corresponding to the interactive window is on the whitelist, it can intercept the interactive window of the APP installation process through the Hook management module, and perform coordinate transformation on the interactive window, and move the interactive window to the virtual screen for display. When the window position management module determines that the application corresponding to the interactive window is not on the whitelist, it starts a thread to patrol the main window of the application created by the electronic device, hides the interactive window on the main screen when patrolling to the interactive window, and performs coordinate transformation on the interactive window, and moves the interactive window to the virtual screen for display.

[0101] In the above embodiment, when the electronic device starts to create an interactive window, it determines whether the application corresponding to the interactive window is in a preset whitelist. If it is not in the whitelist, the Hook interception technology is used to intercept the interactive window and move the display to the virtual screen. If it is in the whitelist, the thread is started to patrol the main window of the application created by the electronic device, and then the interactive window is also moved to the virtual screen for display. This can avoid the abnormal scenario of the interactive window being displayed on the main screen and improve the stability of the automated interactive function. At the same time, when the thread is used to patrol the interactive window, the interactive window can also be hidden on the main screen first, so as to avoid the phenomenon of the interactive window flashing on the main screen, further improving the user experience.

[0102] The above describes in detail an example of the automated interaction method provided by the embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0103] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0104] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0105] The electronic device provided in this embodiment is used to execute the above-mentioned automatic interaction method, and thus can achieve the same effect as the above-mentioned implementation method.

[0106] When integrated, the electronic device may also include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the operation of the electronic device. The storage module may be used to support the execution of program code and data stored in the electronic device. The communication module may be used to support communication between the electronic device and other devices.

[0107] The processing module may be a processor or controller. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0108] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 9 Device with the structure shown.

[0109] For example, Figure 9 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. Taking a PC as an example, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a wireless communication module 150, a display screen 160, etc.

[0110] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0111] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0112] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0113] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0114] In some embodiments, the processor 110 may include one or more interfaces, including an I2C interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface.

[0115] It is understood that the interface connection relationship between the modules illustrated in this embodiment is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0116] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While charging the battery 142, the charging management module 140 can also power the electronic device through the power management module 141.

[0117] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 160, and the wireless communication module 150. In some embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0118] The wireless communication module 150 can provide wireless communication solutions for the electronic device 100, including WLAN (such as Wi-Fi), Bluetooth, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), and other wireless communication technologies. For example, in the embodiment of the present application, the electronic device 100 can establish a Bluetooth connection with a terminal device (such as the wireless headset 100) through the wireless communication module 150.

[0119] The wireless communication module 150 can be one or more devices that integrate at least one communication processing module. The wireless communication module 150 receives electromagnetic waves via an antenna, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 150 can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and convert them into electromagnetic waves for radiation via the antenna.

[0120] Electronic device 100 implements display functionality through a GPU, display screen 160, and an application processor. A GPU is a microprocessor for image processing that connects display screen 160 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0121] The display screen 160 is used to display images, videos, etc. The display screen 160 includes a display panel.

[0122] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0123] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. For example, in an embodiment of the present application, the processor 110 can execute instructions stored in the internal memory 121, and the internal memory 121 can include a program storage area and a data storage area.

[0124] The program storage area can store an operating system and at least one application required for a function (such as a sound playback function or an image playback function). The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or a universal flash storage (UFS).

[0125] The present application also provides a computer-readable storage medium that stores a computer program. When executed by a processor, the computer program causes the processor to perform the automated interaction method of any of the above-described embodiments. The storage medium may include a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0126] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the automated interaction method in the above-mentioned embodiment.

[0127] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the automated interaction method in the above-mentioned method embodiments.

[0128] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0129] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0130] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0131] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An automated interaction method, the method being executed by an electronic device, characterized in that: The method comprises: receiving a first operation input by a user, where the first operation is used to trigger the electronic device to automatically execute a first service; When the first application corresponding to the first service exists in a preset whitelist, starting a thread to patrol the main window of the application created by the electronic device within a first preset time period, the main window of the application including the interactive window corresponding to the execution of the first service; If the interactive window is patrolled and the window coordinates of the interactive window are located on the main screen of the electronic device, the interactive window is moved to the smart window for display. The smart window is located in a partial area of ​​the main screen of the electronic device and is used to display the interactive process of the electronic device automatically executing the first business.

2. The method according to claim 1, characterized in that The step of moving the interactive window to the smart window for display includes: Changing the window coordinates of the interactive window to corresponding coordinates on a virtual screen, wherein the virtual screen is a virtual screen created by the electronic device when receiving the first operation; Displaying the interactive window on the virtual screen; The display interface of the virtual screen is captured and mapped frame by frame to the smart window for display.

3. The method according to claim 2, characterized in that The size of the virtual screen is the same as the size of the display screen of the electronic device.

4. The method according to claim 1, wherein In a case where the interactive window is patrolled and the window coordinates of the interactive window are located on the main screen of the electronic device, the method further includes: The interactive window is hidden on the main screen.

5. The method according to any one of claims 1 to 4, characterized in that After the first operation of receiving user input, the method further includes: Starting a main process for executing the first service; When starting to create the interactive window, suspending the main process; After starting the thread to patrol the main window of the application, the main process is unsuspended.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: If the interactive window is not patrolled, or if the interactive window is patrolled but the window coordinates of the interactive window are not located on the main screen of the electronic device, waiting for a second preset time period; After the second preset time period, the application main window continues to be patrolled.

7. The method according to claim 1, characterized in that The method further comprises: If the first application does not exist in the preset whitelist, intercepting the created interactive window; Changing the window coordinates of the interactive window to corresponding coordinates on the virtual screen; Displaying the interactive window on the virtual screen; The display interface of the virtual screen is captured and mapped frame by frame to the smart window for display.

8. The method according to claim 7, characterized in that The intercepting and creating the interactive window includes: The interactive window is intercepted by using Hook interception technology.

9. The method according to claim 8, characterized in that The preset whitelist stores the identifiers of applications whose windows cannot be intercepted by the Hook interception technology.

10. The method according to claim 1, characterized in that After moving the interactive window to the smart window for display, the method further includes: A click operation of a user is simulated on the interactive window to automatically execute the first service.

11. The method according to claim 1, wherein Before the first operation of receiving user input, the method further includes: Displaying a first interface, wherein the first interface presents a command input control; In response to an instruction input by the user through the instruction input control, displaying an operation control corresponding to triggering execution of the first service; The first operation is received through the operation control.

12. The method according to claim 1, characterized in that The first service includes installing or uninstalling the first application.

13. An electronic device, characterized in that: The electronic device comprises: one or more processors, and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 12.

14. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium comprises instructions, which, when executed on an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 12.

Images