Device control method and apparatus, and electronic device

Abstract

Provided in the present disclosure are a device control method and apparatus, and an electronic device. The device control method comprises: acquiring device control information, which is used for requesting a target device to execute a target event; and determining at least one piece of operation information on the basis of the device control information and a graphical user interface, which is displayed by the target device, and on the basis of the at least one piece of operation information, controlling the target device to execute the target event.

Description

Device control method, device and electronic equipment

[0001] This application claims priority to Chinese Patent Application No. 202311772990.6 filed on December 20, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] Embodiments of the present disclosure relate to a device control method, apparatus, and electronic device. Background Art

[0003] A mobile phone automation system allows a mobile phone to achieve specific steps and goals based on automated scripts. For example, a computer can control a mobile phone to open an application based on a pre-written automated script.

[0004] Currently, automation scripts must be manually written. For example, users can pre-program multiple automation scripts in their electronic devices. Based on the user's selected automation script, the electronic device can control the phone to perform the operations associated with that script. However, manually editing automation scripts is time-consuming and labor-intensive, leading to high implementation costs for mobile phone automation systems.

[0005] Summary of the Invention

[0006] The present disclosure provides a device control method, apparatus, and electronic device.

[0007] In a first aspect, the present disclosure provides a device control method, the method comprising:

[0008] Obtain device control information requesting the target device to execute the target event;

[0009] At least one operation information is determined based on the device control information and the graphical user interface displayed by the target device, and the target device is controlled to execute the target event according to the at least one operation information.

[0010] In a second aspect, the present disclosure provides a device control apparatus, the device control apparatus comprising an acquisition module, a determination module, and a control module, wherein:

[0011] The acquisition module is used to acquire device control information requesting the target device to execute the target event;

[0012] The determining module is configured to determine at least one operation information based on the device control information and a graphical user interface displayed by the target device;

[0013] The control module is configured to control the target device to execute the target event according to the at least one operation information.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: a processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the device control method as described in the first aspect and various possible aspects of the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the device control method as described in the first aspect and various possible aspects of the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0020] FIG2 is a schematic diagram of a flow chart of a device control method provided by an embodiment of the present disclosure;

[0021] FIG3 is a schematic diagram of a process for obtaining device control information provided by an embodiment of the present disclosure;

[0022] FIG4 is a schematic diagram of controlling a target device to execute a target event according to an embodiment of the present disclosure;

[0023] FIG5 is a schematic diagram of controlling a target device to execute a target event according to an embodiment of the present disclosure;

[0024] FIG6 is a schematic diagram of a process for determining i-th operation information provided by an embodiment of the present disclosure;

[0025] FIG7 is a schematic diagram of a process for displaying execution results provided by an embodiment of the present disclosure;

[0026] FIG8 is a process diagram of a device control method provided by an embodiment of the present disclosure;

[0027] FIG9 is a schematic structural diagram of a device control apparatus provided by an embodiment of the present disclosure; and

[0028] FIG10 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0030] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0031] Terminal device: is a device with wireless transceiver function. The terminal device can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted. The terminal device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a vehicle-mounted terminal device, a wireless terminal in self-driving, a wireless terminal device in remote medical, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, a wireless terminal device in smart home, a wearable terminal device, etc. The terminal device involved in the embodiments of the present disclosure can also be referred to as a terminal, user equipment (UE), an access terminal device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote terminal device, a mobile device, a UE terminal device, a wireless communication device, a UE agent or a UE device, etc. Terminal devices can also be fixed or mobile.

[0032] The application scenario of the embodiment of the present disclosure is described below with reference to FIG1 .

[0033] FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure. Please refer to FIG1 , which includes an electronic device and a target device. The electronic device and the target device are connected. Optionally, the electronic device can be connected to the target device based on Bluetooth or based on a network connection, which is not limited by the embodiment of the present disclosure. The display page of the electronic device can be a page automatically controlled by the target device, and the display page can include a video playback control, which can be a control automatically operated by the target device. When the user clicks the control to play the video, the target device can automatically open the video playback application and play the video.

[0034] It should be noted that FIG1 is only an example of an application scenario of the embodiment of the present disclosure, and does not limit the application scenario of the embodiment of the present disclosure.

[0035] In the related art, a mobile phone automation system refers to a system in which a mobile phone can achieve specific steps and goals based on an automation script. The mobile phone can be connected to a computer, and the automation script of the mobile phone can be set in the computer. The user can use the computer to enable the mobile phone to perform automated operations. For example, a user can select an automation script in the computer to open an application, and the computer can control the mobile phone to open the application in the mobile phone. Currently, automation scripts need to be manually edited. For example, a user can pre-edit multiple automation scripts, and the computer can control the mobile phone to perform related operations based on the multiple automation scripts written by the user. However, manually editing automation scripts is time-consuming and labor-intensive. Moreover, when the graphical user interface of the application changes, the automation script also needs to be changed, which results in a high implementation cost of the mobile phone automation system.

[0036] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides a device control method, in which the electronic device can obtain device control information requesting the target device to execute the target event, and obtain the i-th graphical user interface currently displayed by the target device and the step depth of the i-th graphical user interface in the target event, and generate i-th operation information based on the device control information, the step depth and the i-th graphical user interface. The electronic device can generate the i-th control instruction based on the i-th operation information and send the i-th control instruction to the target device, wherein the i-th control instruction is used to make the target device display the i+1-th graphical user interface, where i is 1, 2, ..., in sequence, until the target device completes the target event. In the above method, the electronic device can control the target device to gradually complete the target event based on multiple execution steps, and the electronic device can accurately determine the control instruction associated with each step of the target device based on the graphical user interface and the device control information. Therefore, there is no need to manually write an automation script for the target device, thereby reducing the complexity of the target device performing the automation operation and reducing the cost of the target device performing the automation operation.

[0037] The following detailed description of the technical solution of the present disclosure and how the technical solution of the present disclosure solves the above-mentioned technical problems is provided with specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present disclosure will be described below in conjunction with the accompanying drawings.

[0038] FIG2 is a flow chart of a device control method provided by an embodiment of the present disclosure. Referring to FIG2 , the method may include:

[0039] S201: Obtain device control information for requesting a target device to execute a target event.

[0040] The execution subject of the embodiments of the present disclosure may be an electronic device or a device control device provided in the electronic device. The device control device may be implemented based on software or a combination of software and hardware, which is not limited in the embodiments of the present disclosure.

[0041] Optionally, the target device may be a device connected to the electronic device. For example, the electronic device may be a computer, and the target device may be a mobile phone, tablet computer, or other device connected to the electronic device, which is not limited in the present embodiment.

[0042] The target event may be an event to be executed by the target device. For example, the target event may be opening an application, purchasing a movie ticket, or playing audio. It should be noted that the target event may be any one or more events that can be executed by the target device, and the present disclosure is not limited thereto.

[0043] Optionally, the device control information is used to request the target device to execute a target event. For example, the electronic device may determine the target event to be executed by the target device based on the acquired device control information. For example, if the device control information is "purchase a movie ticket," the target event is "purchase a movie ticket."

[0044] It should be noted that the device control information and the target event correspond to each other. When the electronic device obtains the device control information, the electronic device can determine the target event to be executed by the target device.

[0045] Optionally, the device control information may be text information. For example, the device control information may be a paragraph of text, and the electronic device may determine the target event to be executed by the target device based on the paragraph of text. Optionally, the device control information may be voice information. For example, the device control information may be a paragraph of voice input by the user, and the electronic device may determine the target time to be executed by the target device based on the content of the voice.

[0046] It should be noted that the device control information may also be in any form, and the embodiments of the present disclosure do not limit this.

[0047] Optionally, the electronic device can obtain device control information requesting the target device to execute the target event based on the following feasible implementation method: displaying a dialog box, wherein the dialog box includes an input bar and a send control, and in response to the user's touch operation on the send control, determining the information entered by the user in the input bar as device control information.

[0048] The dialog box may include an input field and a send control. For example, a user may enter information in the input field and click the send control, and the dialog box may display the information entered by the user in the input field. For example, when a user clicks an application (an application related to automation control) in an electronic device, the electronic device may display a dialog box, which may include an input field and a send control. The user may enter any information in the input field, and after the user clicks the send control, the dialog box may display the information, and the electronic device may determine the information entered by the user in the input field as device control information.

[0049] It should be noted that the dialog box may also be displayed on a web page (eg, an electronic device opens a web page related to automated control of a target device, and the web page may include a dialog box), which is not limited in the embodiments of the present disclosure.

[0050] 3 , the process of an electronic device acquiring device control information for requesting a target device to execute a target event will be described below.

[0051] Figure 3 is a schematic diagram of a process for obtaining device control information provided by an embodiment of the present disclosure. Please refer to Figure 3, which includes: an electronic device. Among them, the display page of the electronic device may include a dialog box. The dialog box may include an input bar and a send control. The user can enter "Buy a cup of coffee under 20 yuan" in the input bar and click the send control. The text "Buy a cup of coffee under 20 yuan" can be displayed in the dialog box, and the electronic device can determine that the device control information is: Buy a cup of coffee under 20 yuan. In this way, the user can flexibly enter the device control information in the input box, thereby improving the control flexibility of the target device.

[0052] S202: Determine at least one operation information based on the device control information and the graphical user interface displayed by the target device, and control the target device to execute a target event according to the at least one operation information.

[0053] The graphical user interface (GUI) may be an interface displayed by the target device. For example, the target event may be associated with an application, and thus the GUI displayed by the target device may be the GUI of the application. For example, if the target event is "buy a movie ticket," the graphical user interface displayed by the target device may be related to the movie ticket purchase application.

[0054] It should be noted that the electronic device can obtain the graphical user interface displayed by the target device based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0055] Optionally, the operation information may be information associated with the target event. For example, in actual applications, the target event may be completed based on a single operation (e.g., the target event may be opening a video application) or a multi-step operation (e.g., the target event may be purchasing a movie ticket). The operation information may be information corresponding to the operation steps associated with the target event.

[0056] For example, if the target event is to open a video application software, and the graphical user interface currently displayed on the target device includes the video application software, then the target event can be completed based on a one-step operation. Therefore, the operation information can be text information related to the one-step operation (for example, the operation information can be: click on the video application software).

[0057] For example, if the target event is to play music in a music list, and the graphical user interface currently displayed on the target device includes music application software, then the steps to complete the target event can be: open the music application software - open the music list - play the music in the music list. Therefore, the target event can include 3 operation information, and each operation information can correspond to an operation step. Among them, the first operation information can be: click on the music application software, the second operation information can be: click on the music list, and the third operation information can be: click on the music.

[0058] Optionally, after the electronic device obtains at least one operation information associated with the target event, it may control the target device to execute the target event based on the at least one operation information. For example, if the target event is opening video playback software, and the graphical user interface displayed on the target device includes the video playback software, the electronic device may determine that the operation information is: clicking on the video playback software, and control the target device to execute the operation of clicking on the video playback software.

[0059] For example, if the target event is to open the video playback software, the video playback software is not included in the graphical user interface displayed by the target device. Therefore, the target event requires multiple operation steps to complete. The electronic device can determine that the operation information related to the first operation step is: sliding the page to the right, and control the target device to perform the operation of sliding the page to the right. After the target device completes the operation, the graphical user interface displayed by the target device includes the video playback software. Therefore, the electronic device can determine that the operation information related to the second operation step is: click on the video playback software. The electronic device can control the target device to perform the operation of clicking on the video playback software.

[0060] 4 , the process of the electronic device controlling the target device to execute the target event is described below.

[0061] Figure 4 is a schematic diagram of controlling a target device to execute a target event, according to an embodiment of the present disclosure. Referring to Figure 4 , the apparatus includes an electronic device and a target device. The electronic device displays a page for controlling the automation of the target device, which may include a dialog box containing the text "Open video playback software." The electronic device may determine that the device control information may include "Open a video application."

[0062] As shown in Figure 4, the target device currently displays a GUI that includes a music application and a chat application, but does not include a video application. Therefore, the electronic device can determine that the first operation information is a left-swipe screen and control the target device to perform the left-swipe screen operation. After the target device swipes the screen left, the target device's currently displayed GUI may include a video application.

[0063] As shown in Figure 4 , the electronic device can determine that the second operation information is "clicking a video application" and control the target device to execute the operation of clicking the video application. After the target device executes the operation of clicking the video application, it can display the video. In this way, the electronic device can control the target device to complete the target event based on at least one operation information, eliminating the need for manual pre-written automation scripts. This can reduce the complexity and cost of the target device's automated operations.

[0064] The embodiment of the present disclosure provides a device control method, in which an electronic device can display a dialog box, which can include an input field and a sending control. In response to a user's touch operation on the sending control, the electronic device can determine the information input by the user in the input field as device control information. The electronic device can determine at least one operation information based on the device control information and a graphical user interface displayed by the target device, and control the target device to execute the target event based on the at least one operation information. In this way, the electronic device can control the target device to gradually complete the target event based on the at least one operation information, and the electronic device can accurately determine the control instructions associated with each step of the target device based on the graphical user interface and the device control information. Therefore, there is no need to manually write an automation script for the target device, thereby reducing the complexity of the target device executing the automation operation and reducing the cost of the target device executing the automation operation.

[0065] Based on the embodiment shown in Figure 2, the following, in combination with Figure 5, further illustrates the method of determining at least one operation information based on the device control information and the graphical user interface displayed by the target device in the above-mentioned device control method, and controlling the target device to execute the target event according to the at least one operation information.

[0066] FIG5 is a schematic diagram of a method for controlling a target device to execute a target event according to an embodiment of the present disclosure. Referring to FIG5 , the method flow may include:

[0067] S501: Obtain the i-th graphical user interface currently displayed on the target device and the step depth of the i-th graphical user interface in the target event.

[0068] Where i is associated with the operation step associated with the target event, and the i-th GUI may be the GUI displayed by the target device at step i. For example, if the target event has three operation steps, the first GUI may be the GUI displayed by the target device before executing step 1, the second GUI may be the GUI displayed by the target device after executing step 1 and before executing step 2, and the third GUI may be the GUI displayed by the target device after executing step 2 and before executing step 3.

[0069] For example, in the embodiment shown in Figure 4, the first graphical user interface displayed by the target device may include a music application and a chat application (the graphical user interface displayed by the target device before performing the left-sliding screen operation), and the second graphical user interface displayed by the target device may include a video application (the graphical user interface displayed by the target device after performing the left-sliding screen operation and before clicking on the video application). Moreover, after the target device completes the operation of clicking on the video application (the target device completes all operation steps), the target device may also display the third graphical user interface. When the target device displays this interface, the electronic device can determine that the target device has completed the target event.

[0070] It should be noted that the electronic device can obtain the i-th graphical user interface currently displayed on the target device based on any feasible implementation method, and the embodiment of the present disclosure is not limited to this.

[0071] Optionally, the step depth can be the step depth of the i-th GUI in the target event. For example, if the operation information associated with the target event is 3, it means that the target event requires 3 operation steps to complete. Therefore, the step depths of the GUI displayed by the target device can be 1, 2, and 3. It should be noted that if the target device can display another GUI after completing the target event, the step depth can also be 4.

[0072] For example, in the embodiment shown in Figure 4, the first graphical user interface displayed by the target device may include a music application and a chat application, and the step depth of the graphical user interface may be 1 (i.e., the first step of completing the target event), the second graphical user interface displayed by the target device may include a video application, and the step depth of the graphical user interface may be 2 (i.e., the second step of completing the target event), and the third graphical user interface displayed by the target device may include a video, and the step depth of the graphical user interface may be 3 (i.e., the target event has been completed).

[0073] It should be noted that the electronic device can obtain the step depth of the i-th graphical user interface in the target event based on any feasible implementation method, and the embodiment of the present disclosure is not limited to this.

[0074] S502 : Generate i-th operation information based on the device control information, the step depth, and the i-th graphical user interface.

[0075] Optionally, the electronic device can generate the i-th operation information based on the following feasible implementation method: obtain interface element information in the i-th graphical user interface, and determine the i-th operation information based on device control information, step depth and interface element information in the i-th graphical user interface.

[0076] Among them, the interface element information can be information about each element in the graphical user interface. For example, the interface element information can include information such as the page title, interface description, interface layout, and type of interface element in the graphical application interface. For example, the page title in the graphical user interface can be text information, and the page title can indicate the subject content of the graphical user interface. For example, the interface description information is used to describe the shape of the interface elements in the graphical user interface, the characteristics of the interface elements, and other information, which is not limited in the embodiments of the present disclosure. For example, the interface layout can indicate the layout used by the graphical user interface. For example, the type of interface element can include a text view type, a navigation bar type, etc., which is not limited in the embodiments of the present disclosure.

[0077] It should be noted that the electronic device can obtain the interface element information in the i-th graphical user interface based on any feasible implementation method, and the embodiment of the present disclosure does not limit this. Moreover, the interface element information in the embodiment of the present disclosure can include any information related to the graphical user interface, and the embodiment of the present disclosure does not limit this.

[0078] Optionally, the electronic device determines the i-th operation information based on the device control information, step depth and interface element information in the i-th graphical user interface. Specifically, it can be: sending the device control information, step depth and interface element information in the i-th graphical user interface to the large language model, and receiving the i-th operation information sent by the large language model.

[0079] Optionally, the large language model may be any type of language model (eg, a conversational language model, a question-answering language model, etc.), which is not limited in the embodiments of the present disclosure.

[0080] Optionally, the large language model can play a decision-making role. For example, the electronic device can send interface element information, step depth and device control information to the large language model, and the large language model can determine the steps associated with the target event being executed this time. For example, in the embodiment shown in Figure 4, if the electronic device sends the interface element information, step depth and device control information of the first graphical user interface to the large language model, the large language model can determine that the step being executed this time is the first step in the target event, and then can determine that the first operation information is: sliding the screen to the left. If the electronic device sends the interface element information, step depth and device control information of the second graphical user interface to the large language model, the large language model can determine that the step being executed this time is the second step in the target event, and then can determine that the second operation information is: clicking on the video application. If the electronic device sends the interface element information, step depth and device control information of the third graphical user interface to the large language model, the large language model can determine that the target event has been completed based on the above information.

[0081] 6 , the process of the electronic device determining the i-th operation information will be described below.

[0082] Figure 6 is a schematic diagram of a process for determining the i-th operation information provided by an embodiment of the present disclosure. Please refer to Figure 6, which includes: a target device, an electronic device, and a large language model. Among them, the electronic device can obtain the i-th interface element information of the i-th graphical user interface currently displayed by the target device, and the electronic device can send the i-th step depth, device control information and i-th interface element information of the i-th step to the large language model. The large language model can analyze the above information, and then determine the i-th operation information associated with the operation that the target device currently needs to perform, and send the i-th operation information to the electronic device. In this way, the operation performed by the target device each time can be decided by the large language model. Therefore, there is no need to manually write related automation scripts, which can reduce the complexity and cost of the automated operation of the target device.

[0083] S503: Generate an i-th control instruction based on the i-th operation information, and send the i-th control instruction to the target device.

[0084] The i-th control instruction is used to cause the target device to display the i+1-th graphical user interface. For example, after the electronic device generates the i-th control instruction, it can send the control instruction to the target device. The graphical user interface currently displayed on the target device is the i-th graphical user interface. After the target device executes the i-th control instruction, the target device can display the i+1-th graphical user interface.

[0085] Optionally, the (i+1)th GUI is associated with the (i)th GUI and the (i)th control instruction. For example, in the embodiment shown in FIG4 , when i is 1, the (i)th GUI may include a music application and a chat application. If the (i)th control instruction is an instruction to slide the screen to the left, the (i+1)th GUI may be the interface after the (i)th GUI is slid to the left. If the (i)th control instruction is an instruction to click on the music application, the (i+1)th GUI may be the interface of the music application.

[0086] It should be noted that, during the process of the target device executing the target event, the next GUI of the i-th GUI may be the i+1-th GUI. For example, in the embodiment shown in FIG4 , during the process of the target device executing the event of opening a video application, after the target device displays the first GUI, when the target device performs an operation of sliding the screen to the left, the target device may display the next GUI of the first GUI (i.e., the target device may display the second GUI).

[0087] Here, i is 1, 2, ..., until the target device completes the target event. For example, if the target event has three steps, i can be 1, 2, and 3. If the target device can display a new graphical user interface after completing the three steps, i can be 4. When i is 4, the electronic device can determine that the target device has completed the target event based on the graphical user interface displayed by the target device.

[0088] Optionally, the i-th control instruction may be a control instruction corresponding to the i-th operation information. For example, the control instruction is an instruction that can be executed by the target device, and the control instruction may be a script or code. For example, the i-th operation information may be text information, and the i-th control instruction associated with the i-th operation information may include code. For example, if the i-th operation information is to click on a video application, the function of the i-th control instruction associated with the i-th operation information is to perform a touch operation on the location where the video application is located.

[0089] Optionally, the electronic device may generate the i-th control instruction based on the following feasible implementation method: obtaining a preset control instruction script, adding the i-th operation information to the preset control instruction script, and obtaining the i-th control instruction.

[0090] Optionally, the preset control instruction script can be a universal script. For example, the multiple codes in the preset control instruction script can be pre-written codes, and the preset control instruction script can include a reserved blank field. The electronic device can add the i-th operation information to the position of the reserved blank field in the preset control instruction script, and thus obtain the i-th control instruction corresponding to the i-th operation information. For example, if the i-th operation information is to click on the video application, the electronic device can add the i-th operation information to the position of the reserved field of the control instruction script, and thus obtain the i-th control instruction. When the target device executes the i-th control instruction, it can perform a touch operation on the position of the video application. In this way, the user does not need to write the associated code for each control instruction, thereby reducing the complexity of the automated control of the target device and reducing the cost of automated control of the target device.

[0091] It should be noted that the electronic device may also generate the i-th control instruction associated with the i-th operation information based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0092] Optionally, after the electronic device sends the i-th control instruction to the target device, it can also determine the execution result of the i-th control instruction sent by the target device. After the electronic device determines the execution result of the i-th control instruction sent by the target device, the above method also includes: determining the i-th execution result associated with the i-th control instruction, displaying the execution result of the i-th control instruction in a dialog box, and adding the i-th execution result to the device control information.

[0093] The i-th execution result may be the result after the target device executes the i-th control instruction. For example, the i-th execution result may be text information. For example, if the i-th control instruction may be a control instruction for clicking a video application, the i-th execution result associated with the i-th control instruction may be text: the video application has been opened; if the i-th control instruction may be a control instruction for sliding the screen to the left, the i-th execution result associated with the i-th control instruction may be voice: the screen has been slid to the left.

[0094] It should be noted that the i-th execution result can be any form of information such as text information, voice information, etc., and the embodiments of the present disclosure do not limit this. In addition, the electronic device can determine the i-th execution result associated with the i-th control instruction based on any feasible implementation method (for example, pre-setting multiple execution results associated with each control instruction, or determining the execution results associated with the control instruction based on a large language model), and the embodiments of the present disclosure do not limit this.

[0095] Optionally, after the electronic device determines the i-th execution result associated with the i-th control instruction, it may display the execution result of the i-th control instruction in a dialog box. For example, if the i-th control instruction is a control instruction to click on a video application, then when the target device completes executing the i-th control instruction, the electronic device may display the text "Video application opened" in the dialog box. In this way, the user can determine the execution steps of the target event based on the content in the dialog box, thereby improving the user experience.

[0096] Optionally, after the electronic device determines the i-th execution result associated with the i-th control instruction, it can also add the i-th execution result to the device information. In this way, the richness of the device control information can be improved, and the accuracy of the i+1-th operation information can be improved, thereby improving the accuracy of the automated control of the target device.

[0097] It should be noted that the device control information may also include description information of the application currently used by the target device, the decision result required to be output, information on whether the target event is completed, decision basis, etc., which is not limited in the present disclosure.

[0098] The following describes the process of displaying the i-th execution result in the dialog box with reference to FIG. 7 .

[0099] Figure 7 is a schematic diagram of a process for displaying execution results provided by an embodiment of the present disclosure. Referring to Figure 7 , a dialog box may be included. The dialog box may include an input field and a send control. A user may enter the text "Open Video App" in the input field and click the send control, and the dialog box may display the text "Open Video App."

[0100] Referring to Figure 7 , an electronic device (not shown in Figure 7 ) can control a target device (not shown in Figure 7 ) to open a video application. After the electronic device controls the target device to slide the screen to the left, a dialog box can display the text: "Screen slid to the left." After the electronic device controls the target device to click on the video application, a dialog box can display the text: "Video application opened." In this way, in scenarios where repeated automatic testing of the target device's application is performed, the user can obtain the execution progress of the target event in a timely and accurate manner based on the text in the dialog box, thereby improving the user experience.

[0101] The disclosed embodiments provide a method for controlling a target device to execute a target event. The electronic device can obtain the i-th graphical user interface currently displayed on the target device and the step depth of the i-th graphical user interface in the target event. Based on the device control information, the step depth, and the i-th graphical user interface, the electronic device generates the i-th operation information. Based on the i-th operation information, the electronic device generates the i-th control instruction and sends the i-th control instruction to the target device. In this way, the electronic device can accurately determine the control instruction associated with each step of the target event, thereby improving the accuracy of the target device's execution of the target event and the accuracy of the target device's automated control.

[0102] Based on any of the above embodiments, the process of the above device control method will be described in detail below with reference to FIG8 .

[0103] FIG8 is a process diagram of a device control method provided by an embodiment of the present disclosure. Referring to FIG8 , the method includes: an electronic device and a target device. The electronic device and the target device are connected. The display page of the electronic device is an automated control page for the target device. The display page may include a dialog box, which may include text such as "Open a video application." The electronic device may determine that device control information 1 is "Open a video application." The first graphical user interface currently displayed on the target device may include a music application and a chat application.

[0104] Referring to Figure 8 , the electronic device may send step depth 1 (step 1), interface element information 1 (interface element information in the first graphical user interface), and device control information 1 to the large language model. The large language model may send the electronic device the operation information for step 1: swipe the screen left. Based on this operation information, the electronic device may control the target device to perform the swipe left operation.

[0105] Please refer to Figure 8. After the electronic device controls the target device to perform the operation of sliding the screen to the left, the dialog box of the electronic device may include text: open the video application, the screen has been slid to the left, and the electronic device may determine that the device control information 2 is: open the video application, slide the screen to the left. The second graphical user interface currently displayed by the target device may include a video application. The electronic device may send step depth 2 (step 2), interface element information 2 (interface element information in the second graphical user interface) and device control information 2 to the large language model.

[0106] Referring to FIG8 , the large language model may send the operation information of step 2 to the electronic device: click on the video application. Based on this operation information, the electronic device may control the target device to perform a single touch operation at the location of the video application. After the electronic device controls the target device to click on the video application, the dialog box of the electronic device may include the text "Open video application", "Screen has been swiped to the left", and "Video application has been opened". The graphical user interface currently displayed on the target device may include a video that automatically plays after the video application is opened.

[0107] Optionally, in the embodiment shown in Figure 8, the electronic device can also send step depth 3 (step 3), device control information 3 (open video application, swiped the screen to the left, opened video application) and interface element information 3 (interface element information of the third graphical user interface) to the large language model. The large language model can determine that the target event has been completed, and then send operation information to stop the operation to the electronic device, and the electronic device can stop controlling the target device.

[0108] In this way, the electronic device can control the target device to gradually complete the target event based on multiple execution steps, and the electronic device can accurately determine the control instructions associated with each step of the target device based on the graphical user interface and device control information. Therefore, there is no need to manually write an automation script for the target device, thereby reducing the complexity of the target device performing automated operations and reducing the cost of the target device performing automated operations.

[0109] FIG9 is a schematic diagram of the structure of a device control apparatus provided by an embodiment of the present disclosure. Referring to FIG9 , the device control apparatus 900 includes an acquisition module 901, a determination module 902, and a control module 903, wherein:

[0110] The acquisition module 901 is used to acquire device control information requesting the target device to execute the target event;

[0111] The determining module 902 is configured to determine at least one operation information based on the device control information and the graphical user interface displayed by the target device;

[0112] The control module 903 is configured to control the target device to execute the target event according to the at least one operation information.

[0113] According to one or more embodiments of the present disclosure, the determining module 902 and the controlling module 903 are specifically configured to:

[0114] Obtaining the i-th graphical user interface currently displayed on the target device and the step depth of the i-th graphical user interface in the target event;

[0115] generating, based on the device control information, the step depth, and the i-th graphical user interface, i-th operation information;

[0116] Based on the i-th operation information, generate an i-th control instruction, and send the i-th control instruction to the target device, where the i-th control instruction is used to cause the target device to display an i+1-th graphical user interface;

[0117] Here, the value of i is 1, 2, ... in sequence, until the target device completes executing the target event.

[0118] According to one or more embodiments of the present disclosure, the determining module 902 is specifically configured to:

[0119] Obtaining interface element information in the i-th graphical user interface;

[0120] The i-th operation information is determined based on the device control information, the step depth, and the interface element information in the i-th graphical user interface.

[0121] According to one or more embodiments of the present disclosure, the determining module 902 is specifically configured to:

[0122] Sending the device control information, the step depth, and the interface element information in the i-th graphical user interface to the large language model;

[0123] Receive the i-th operation information sent by the large language model.

[0124] According to one or more embodiments of the present disclosure, the control module 903 is specifically configured to:

[0125] Get the preset control command script;

[0126] The i-th operation information is added to the preset control instruction script to obtain the i-th control instruction.

[0127] According to one or more embodiments of the present disclosure, the acquisition module 901 is specifically configured to:

[0128] Display a dialog box, wherein the dialog box includes an input field and a send control;

[0129] In response to a user's touch operation on the sending control, the information input by the user in the input field is determined as the device control information.

[0130] According to one or more embodiments of the present disclosure, the control module 903 is further configured to:

[0131] Determine the i-th execution result associated with the i-th control instruction;

[0132] The execution result of the i-th control instruction is displayed in a dialog box, and the i-th execution result is added to the device control information.

[0133] The device control device provided in this embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0134] FIG10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Please refer to FIG10 , which shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing an embodiment of the present disclosure. The electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG10 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0135] As shown in Figure 10, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the electronic device 1000 are also stored in the RAM 1003. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0136] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. Although FIG10 shows an electronic device 1000 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0137] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0138] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0139] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0140] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0141] An embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, various methods that may be involved in the above embodiments are implemented.

[0142] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements various possible methods involved in the above embodiments when executed by a processor.

[0143] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0146] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0147] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0149] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0150] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0151] For example, in response to receiving an active request from the user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thereby, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server or storage medium that performs the operation of the technical solution of the present disclosure based on the prompt message. As an optional but non-limiting implementation method, in response to receiving an active request from the user, the method of sending a prompt message to the user can be, for example, a pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0152] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0153] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws and regulations. Data may include information, parameters and messages, such as flow switching indication information.

[0154] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0155] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0156] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A device control method, comprising: Obtain device control information requesting the target device to execute the target event; At least one operation information is determined based on the device control information and the graphical user interface displayed by the target device, and the target device is controlled to execute the target event according to the at least one operation information.

2. The method according to claim 1, wherein: The determining at least one operation information based on the device control information and the graphical user interface displayed by the target device, and controlling the target device to execute the target event according to the at least one operation information, comprises: Obtaining the i-th graphical user interface currently displayed on the target device and the step depth of the i-th graphical user interface in the target event; generating, based on the device control information, the step depth and the i-th graphical user interface, i-th operation information; Based on the i-th operation information, generate an i-th control instruction, and send the i-th control instruction to the target device, wherein the i-th control instruction is used to make the target device display an i+1-th graphical user interface; Here, i is 1, 2, ... in sequence until the target device completes the target event.

3. The method according to claim 2, wherein: The generating, based on the device control information, the step depth and the i-th graphical user interface, the i-th operation information comprises: Obtaining interface element information in the i-th graphical user interface; The i-th operation information is determined based on the device control information, the step depth, and the interface element information in the i-th graphical user interface.

4. The method according to claim 3, wherein: The determining the i-th operation information based on the device control information, the step depth, and the interface element information in the i-th graphical user interface includes: Sending the device control information, the step depth, and the interface element information in the i-th graphical user interface to the large language model; Receive the i-th operation information sent by the large language model.

5. The method according to any one of claims 2 to 4, wherein: The generating the i-th control instruction based on the i-th operation information includes: Get the preset control command script; The i-th operation information is added to the preset control instruction script to obtain the i-th control instruction.

6. The method according to any one of claims 2 to 5, wherein: The obtaining of device control information requesting the target device to execute the target event includes: Display a dialog box, wherein the dialog box includes an input field and a send control; In response to a user's touch operation on the sending control, the information input by the user in the input field is determined as the device control information.

7. The method according to any one of claims 2 to 6, wherein: After sending the i-th control instruction to the target device, the method further includes: Determining an i-th execution result associated with the i-th control instruction; The execution result of the i-th control instruction is displayed in a dialog box, and the i-th execution result is added to the device control information.

8. A device control apparatus, comprising an acquisition module, a determination module and a control module, wherein: The acquisition module is configured to acquire device control information requesting the target device to execute the target event; The determination module is configured to determine at least one operation information based on the device control information and the graphical user interface displayed by the target device; The control module is configured to control the target device to execute the target event according to the at least one operation information.

9. An electronic device comprising a processor and a memory, wherein: The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the device control method according to any one of claims 1 to 7.

10. A computer-readable storage medium stores computer-executable instructions, wherein: When the processor executes the computer-executable instruction, the device control method according to any one of claims 1 to 7 is implemented.

Images