Intelligent interaction method and electronic equipment

Abstract

The invention provides an intelligent interaction method and electronic equipment. The intelligent assistant can directly interact with the application, and triggers the application to execute the voice command based on the acquired page data, so that the interaction process is simplified, and the communication efficiency is improved. The method comprises the steps that an intelligent assistant in the electronic equipment receives a first voice command of a user, the first voice command carries information of a first application, and the first application is an application currently displayed by the electronic equipment; and the intelligent assistant requests the first application to obtain the first page data according to the first voice command. And the intelligent assistant triggers the first application to execute a first event corresponding to the first voice command according to the acquired first page data.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an intelligent interaction method and electronic device. Background Art

[0002] With the development of terminal technology, electronic devices are equipped with intelligent assistants. When it is inconvenient for users to operate the electronic device directly, they can use intelligent assistants to control the electronic device by voice. For example, when it is inconvenient for users to operate the screen of the smart cockpit by hand while driving, users can use the intelligent assistant of the smart cockpit to instruct the smart cockpit to launch the navigation application through voice commands.

[0003] During intelligent interaction, the smart assistant relies on the screenshot function of the accessibility service configured on the electronic device to obtain a screenshot. Through image analysis, it obtains the interface information of the screenshot to determine the location of the control indicated by the voice command. The smart assistant then sends this control location to the accessibility service, which triggers the application to execute the event corresponding to the control.

[0004] It can be seen that the current implementation of the intelligent interaction process relies on barrier-free services, the interaction process is cumbersome, the image analysis process is complex, and the power consumption overhead is large. Summary of the Invention

[0005] To solve the above technical problems, this application provides an intelligent interaction method and electronic device. The technical solution provided by this application enables the intelligent assistant to directly interact with the application and trigger the application to execute voice commands based on the acquired page data, thus simplifying the interaction process and improving communication efficiency.

[0006] In order to achieve the above technical objectives, this application provides the following technical solutions:

[0007] In a first aspect, an intelligent interaction method is provided, comprising: an intelligent assistant in an electronic device receiving a first voice command from a user, the first voice command carrying information about a first application, the first application being the application currently displayed on the electronic device; the intelligent assistant requesting, based on the first voice command, from the first application to obtain first page data; and the intelligent assistant triggering, based on the obtained first page data, the first application to execute a first event corresponding to the first voice command.

[0008] It should be understood that with existing technologies, smart assistants cannot obtain application page data, but instead need to infer application data through screenshots; this method has many drawbacks. In the method provided in this application, smart assistants can interact directly with applications and obtain page data that is more suitable for voice command execution, avoiding the acquisition of redundant information. In addition, the intelligent interaction process does not need to rely on barrier-free services, avoiding unnecessary memory and power consumption, and improving the efficiency of intelligent interaction.

[0009] According to the first aspect, the first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via inter-process communication (IPC).

[0010] The control tree data includes, for example, the controls included in the current display interface, control identifiers, the hierarchical relationship of each control, the display position of the control, the display size of the control, and other data.

[0011] According to the first aspect, or any implementation of the first aspect above, the intelligent assistant triggers the first application to execute a first event corresponding to the first voice command based on the acquired first page data, including: the intelligent assistant acquires, based on the acquired first page data, a first control in the first application instructed to operate by the first voice command. The intelligent assistant sends a first operation instruction to the first application, and the first operation instruction triggers the first application to operate the first control to execute the first event corresponding to the first voice command.

[0012] The first control may be, for example, a control included in the current display interface of the first application. For example, if the first application is a chat application and the current display interface is the chat interface, the first control may be, for example, an input box or a send control in the chat interface. The number of first controls may be one or more.

[0013] For example, the first voice command carries information about the first application, and the first voice command may explicitly record the name of a certain application. In some scenarios, the first voice command may be, for example, "Xiaoyi, transfer the Changlian video call to the Huawei Smart Screen," where Changlian is the first application.

[0014] For example, the first voice command carries information about the first application, and the first voice command may implicitly record the name of a certain application. In some scenarios, the first voice command may be, for example, "Xiaoyi, what other movies has the protagonist of this movie starred in?"; wherein, the intelligent assistant may use the video application (such as Huawei Video) that is currently playing the movie as the first application to execute the method provided by the first aspect of this application. The first voice command may also be, for example, "Order me some fried chicken."

[0015] For example, when the electronic device is displaying a chat application (such as the first application), the intelligent assistant receives the user's first voice command "Send XX message". In response to the first voice command, the intelligent assistant requests the chat application to obtain the first page data. Based on the first page data, the intelligent assistant can determine that the first control indicated by the first voice command is an input box, and the control command is to enter XX message in the input box and send it. Then, the intelligent assistant can send an operation instruction to the chat application, which is used to instruct the chat application to execute the control command. Among them, the control command is, for example, a command for implementing the first event.

[0016] This allows the smart assistant to interact directly with the app, obtaining page data that is more suitable for voice command execution and avoiding the acquisition of redundant information. Furthermore, the smart interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption, and improving the efficiency of smart interaction.

[0017] It should be understood that the second voice command or other voice commands exemplified below can also be understood by analogy, and will not be elaborated on here.

[0018] According to the first aspect, or any implementation of the first aspect above, the method further includes: the electronic device displaying a page of the second application. The intelligent assistant receives a second voice command from the user; wherein the second voice command carries information about the third application. The intelligent assistant obtains second page data of the second application based on the second voice command; and the intelligent assistant triggers the third application to execute a second event based on the second page data, where the second event is related to the second page data.

[0019] In this way, the smart assistant can obtain the second page data of the second application through IPC communication with the second application currently being displayed.

[0020] According to the first aspect, or any implementation of the first aspect above, the intelligent assistant triggers the third application to execute the second event based on the second page data, including: the intelligent assistant instructs the third application to execute the second event in the background based on the second page data. Alternatively, the intelligent assistant triggers the electronic device to display the third application in response to the second voice command. The intelligent assistant instructs the third application to execute the second event based on the second page data.

[0021] In this way, when multiple applications are involved in the voice command, the intelligent assistant can also realize direct IPC communication with multiple applications, triggering corresponding application execution events, thereby improving communication efficiency.

[0022] According to the first aspect, or any implementation of the first aspect above, the intelligent assistant triggers a third application to execute a second event based on the second page data, including: the intelligent assistant requests the third application to obtain the third page data. The intelligent assistant sends a second operation instruction to the third application based on the second page data and the third page data to trigger the third application to execute the second event.

[0023] In this way, the intelligent assistant obtains the page data of the currently displayed application based on the voice command, and triggers other applications to run in the foreground based on this page data to obtain the page data of other applications. In this way, the intelligent assistant combines the page data obtained multiple times to trigger the execution event of the corresponding application. In this way, through direct interaction between applications, a highly efficient intelligent interaction process is achieved.

[0024] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to a page change of the first application, the intelligent assistant obtains the updated first page data of the first application.

[0025] In this way, the intelligent assistant can determine the execution status of the first voice command based on the latest page display of the first application, or continue to execute the first voice command. Alternatively, after receiving other voice commands, it can also execute the corresponding voice command based on the updated first page data.

[0026] According to the first aspect, or any implementation of the first aspect above, the method also includes: if the user's third voice command is not received within a preset time, the intelligent assistant stops obtaining the first page data from the first application.

[0027] For example, if the smart assistant does not receive any more user voice commands within a preset time, it may automatically exit.

[0028] In this way, the smart assistant can automatically stop the intelligent interaction process, avoiding long waits for user voice commands and increasing device power consumption.

[0029] According to the first aspect, or any implementation of the first aspect above, before the intelligent assistant requests the first application to obtain the first page data based on the first voice command, the method further includes: the intelligent assistant requests the window management system (WMS) to obtain the currently displayed first window based on the first voice command, where the first window is used to display the first application. Requesting the first application to obtain the first page data includes: the intelligent assistant obtaining the first page data through an interface of the first window.

[0030] In some examples, the electronic device is configured with a WMS, which is used to manage and control the display and interaction of multiple windows on the screen of the electronic device. Then, based on the first voice command, the intelligent assistant requests the WMS to obtain the currently displayed first window, which is used to display the first application. The first window is, for example, the focus window. The WMS can feedback the interface of the first window to the intelligent assistant, and the intelligent assistant obtains the first page data through the interface of the first window.

[0031] Generally, a user controls the currently displayed first application through a voice command, or instructs to start the first application, so as to control the first application.

[0032] In this way, the smart assistant can obtain the first application currently being displayed through WMS, thereby obtaining the first page data of the first application.

[0033] In a second aspect, an intelligent interaction method is provided, comprising: an intelligent assistant in an electronic device receiving a first voice command from a user, the first voice command carrying first information about a first application and second information about a second application, where the first application is the application currently displayed on the electronic device. Based on the first voice command, the intelligent assistant requests the first application to obtain first page data. Based on the first page data, the intelligent assistant triggers the second application to execute a first event corresponding to the first voice command; wherein the first event is related to the first page data.

[0034] The first information of the first application carried in the first voice command may be information directly indicating the first application or information associated with the first application. The second information of the second application carried in the first voice command may be information directly indicating the second application or information associated with the second application.

[0035] In this way, the user can control multiple applications through the first voice command. Alternatively, the user can control other applications based on the page data of the currently displayed application.

[0036] According to the second aspect, the first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via inter-process communication IPC.

[0037] In this way, the intelligent assistant can directly realize IPC communication with the first application without going through the accessibility service, thereby improving the efficiency of intelligent interaction.

[0038] According to the second aspect, or any implementation of the second aspect above, the intelligent assistant triggers the second application to execute the first event corresponding to the first voice command based on the first page data, including: the intelligent assistant instructs the second application to execute the first event in the background based on the first page data. Alternatively, the intelligent assistant triggers the electronic device to display the second application in response to the first voice command. The intelligent assistant instructs the second application to execute the first event based on the first page data.

[0039] According to the second aspect, or any implementation of the second aspect, the intelligent assistant triggers the second application to execute the first event based on the first page data, including: the intelligent assistant requests the second application to obtain the second page data. The intelligent assistant sends a second operation instruction to the second application based on the first page data and the second page data to trigger the second application to execute the first event.

[0040] According to the second aspect, or any implementation of the second aspect above, the method further includes: in response to a page change of the first application, the intelligent assistant obtains the updated first page data of the first application.

[0041] According to the second aspect, or any implementation of the second aspect above, the method also includes: if the user's second voice command is not received within a preset time, the intelligent assistant stops obtaining the first page data from the first application.

[0042] According to the second aspect, or any implementation of the second aspect above, before the intelligent assistant requests the first application to obtain the first page data based on the first voice command, the method further includes: the intelligent assistant requests the window management system (WMS) to obtain the currently displayed first window based on the first voice command, where the first window is used to display the first application. Requesting the first application to obtain the first page data includes: the intelligent assistant obtaining the first page data through an interface of the first window.

[0043] In a third aspect, an intelligent interaction method is provided, which is applied to an electronic device, the electronic device including an intelligent assistant and a first application. The method includes: the intelligent assistant receives a first voice command from a user, the first voice command carrying information about a first application, and the first application is the application currently displayed on the electronic device. The intelligent assistant sends a first page data acquisition request to the first application based on the first voice command. In response to the received first page data acquisition request, the first application sends the first page data to the intelligent assistant. The intelligent assistant triggers the first application to execute a first event corresponding to the first voice command based on the acquired first page data.

[0044] In this way, IPC communication between the smart assistant and the first application is achieved through the first page data request, which can obtain the first page data that is more suitable for voice command execution and avoid the acquisition of redundant information. In addition, the intelligent interaction process does not need to rely on accessibility services, avoiding unnecessary memory and power consumption, and improving the efficiency of intelligent interaction.

[0045] According to a third aspect, the first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via inter-process communication IPC.

[0046] According to the third aspect, or any implementation of the third aspect above, the intelligent assistant triggers the first application to execute a first event corresponding to the first voice command based on the acquired first page data, including: the intelligent assistant acquires, based on the acquired first page data, a first control in the first application instructed to operate by the first voice command. The intelligent assistant sends a first operation instruction to the first application. In response to the first operation instruction, the first application operates the first control to execute the first event corresponding to the first voice command.

[0047] According to the third aspect, or any implementation of the third aspect above, the method further includes: the electronic device displays a page of the second application. The intelligent assistant receives a second voice command from the user; wherein the second voice command carries information about the third application. The intelligent assistant sends a request to the second application to obtain the second page data based on the second voice command. In response to the received request to obtain the second page data, the second application sends the currently displayed second page data to the intelligent assistant. The intelligent assistant triggers the third application to execute a second event based on the second page data.

[0048] In this way, the intelligent assistant can also obtain the second page data through IPC communication with the second application, thereby improving the efficiency of intelligent interaction.

[0049] According to the third aspect, or any implementation of the third aspect above, the intelligent assistant triggers the third application to execute the second event based on the second page data, including: the intelligent assistant sends a first execution command to the third application based on the second page data. In response to the received first execution command, the third application executes the second event in the background. Alternatively, the intelligent assistant triggers the electronic device to display the third application in response to the second voice command. The intelligent assistant sends a second execution command to the third application based on the second page data. In response to the received second execution command, the third application executes the second event.

[0050] According to the third aspect, or any implementation of the third aspect above, the intelligent assistant sends a second execution command to a third application based on the second page data; and in response to the received second execution command, the third application executes a second event, including: the intelligent assistant sends a third page data acquisition request to the third application. In response to the received third page data acquisition request, the third application sends the currently displayed third page data to the intelligent assistant. The intelligent assistant sends a second operation instruction to the third application based on the second page data and the third page data. In response to the second operation instruction, the third application executes the second event.

[0051] In this way, the intelligent assistant can also obtain third-page data through IPC communication with the third application, thereby improving the efficiency of intelligent interaction.

[0052] According to the third aspect, or any implementation of the third aspect above, the method further includes: in response to a page change of the first application, the intelligent assistant obtains the updated first page data of the first application.

[0053] According to the third aspect, or any implementation of the third aspect above, the method also includes: if the user's third voice command is not received within a preset time, the intelligent assistant stops obtaining the first page data from the first application.

[0054] According to the third aspect, or any implementation of the third aspect above, the electronic device further includes a window management system WMS, and the intelligent assistant sends a first page data acquisition request to the first application based on the first voice command; in response to the received first page data acquisition request, the first application sends the currently displayed first page data to the intelligent assistant, including: the intelligent assistant requests the WMS to obtain the currently displayed first window based on the first voice command, where the first window is used to display the first application. The intelligent assistant obtains the first page data of the first application through an interface of the first window.

[0055] The technical effects corresponding to the third aspect and any implementation method of the third aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0056] In a fourth aspect, an intelligent interaction method is provided, which is applied to an electronic device, wherein the electronic device includes an intelligent assistant, a first application, and a second application. The method includes: the intelligent assistant receives a first voice command from a user, wherein the first voice command carries first information of the first application and second information of the second application, and the first application is the application currently displayed on the electronic device. The intelligent assistant sends a first page data acquisition request to the first application based on the first voice command. In response to the received first page data acquisition request, the first application sends the currently displayed first page data to the intelligent assistant. The intelligent assistant triggers the second application to execute a first event corresponding to the first voice command based on the acquired first page data, wherein the first event is related to the first page data.

[0057] In this way, IPC communication between the smart assistant and the first application is achieved through the first page data request, which can obtain the first page data that is more suitable for voice command execution and avoid the acquisition of redundant information. In addition, the intelligent interaction process does not need to rely on accessibility services, avoiding unnecessary memory and power consumption, and improving the efficiency of intelligent interaction.

[0058] Moreover, when the voice command involves multiple applications, the intelligent assistant can also trigger the execution events of the corresponding applications, thereby improving communication efficiency.

[0059] According to a fourth aspect, the first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via inter-process communication IPC.

[0060] According to the fourth aspect, or any implementation of the fourth aspect above, the intelligent assistant triggers the second application to execute the first event corresponding to the first voice command based on the acquired first page data, including: the intelligent assistant sends the first execution command to the second application based on the first page data. In response to the received first execution command, the second application executes the first event in the background. Alternatively, the intelligent assistant triggers the electronic device to display the second application in response to the first voice command. The intelligent assistant sends the second execution command to the second application based on the first page data. In response to the received second execution command, the second application executes the first event.

[0061] According to the fourth aspect, or any implementation of the fourth aspect above, the intelligent assistant triggers the second application to execute the first event based on the first page data, including: the intelligent assistant sends a second page data acquisition request to the second application. In response to the received second page data acquisition request, the second application sends the currently displayed second page data to the intelligent assistant. The intelligent assistant sends a second operation instruction to the second application based on the first page data and the second page data. In response to the second operation instruction, the second application executes the first event.

[0062] According to the fourth aspect, or any implementation of the fourth aspect above, the method further includes: in response to a page change of the first application, the intelligent assistant obtains the updated first page data of the first application.

[0063] According to the fourth aspect, or any implementation of the fourth aspect above, the method also includes: if the user's second voice command is not received within a preset time, the intelligent assistant stops obtaining the first page data from the first application.

[0064] According to the fourth aspect, or any implementation of the fourth aspect above, before the intelligent assistant requests the first application to obtain the first page data based on the first voice command, the method further includes: the intelligent assistant requests the window management system (WMS) to obtain the currently displayed first window based on the first voice command, where the first window is used to display the first application. Requesting the first application to obtain the first page data includes: the intelligent assistant obtaining the first page data through an interface of the first window.

[0065] The technical effects corresponding to the fourth aspect and any implementation method of the fourth aspect can be found in the technical effects corresponding to the above-mentioned second aspect and any implementation method of the second aspect, and will not be repeated here.

[0066] In a fifth aspect, an electronic device is provided. The electronic device includes: a processor and a memory. The memory is coupled to the processor, and the memory is used to store computer program code. The computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes: an intelligent assistant in the electronic device receives a first voice command from the user, and the first voice command carries information about a first application, and the first application is the application currently displayed by the electronic device. Based on the first voice command, the intelligent assistant requests the first application to obtain first page data. Based on the obtained first page data, the intelligent assistant triggers the first application to execute a first event corresponding to the first voice command.

[0067] According to the fifth aspect, the first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via inter-process communication IPC.

[0068] According to the fifth aspect, or any implementation of the fifth aspect above, the intelligent assistant triggers the first application to execute a first event corresponding to the first voice command based on the acquired first page data, including: the intelligent assistant acquires, based on the acquired first page data, a first control in the first application instructed to operate by the first voice command. The intelligent assistant sends a first operation instruction to the first application, and the first operation instruction triggers the first application to operate the first control to execute the first event corresponding to the first voice command.

[0069] According to the fifth aspect, or any implementation of the fifth aspect above, when the processor reads the computer instructions from the memory, it also causes the electronic device to execute: the electronic device displays a page of a second application. The intelligent assistant receives a second voice command from the user; wherein the second voice command carries information about a third application. The intelligent assistant obtains second page data of the second application based on the second voice command; and based on the second page data, the intelligent assistant triggers the third application to execute a second event, where the second event is related to the second page data.

[0070] According to the fifth aspect, or any implementation of the fifth aspect above, the intelligent assistant triggers the third application to execute the second event based on the second page data, including: the intelligent assistant instructs the third application to execute the second event in the background based on the second page data. Alternatively, the intelligent assistant triggers the electronic device to display the third application in response to the second voice command. The intelligent assistant instructs the third application to execute the second event based on the second page data.

[0071] According to the fifth aspect, or any implementation of the fifth aspect, the intelligent assistant triggers the third application to execute the second event based on the second page data, including: the intelligent assistant requests the third application to obtain the third page data. The intelligent assistant sends a second operation instruction to the third application based on the second page data and the third page data to trigger the third application to execute the second event.

[0072] According to the fifth aspect, or any implementation of the above fifth aspect, when the processor reads computer instructions from the memory, it also enables the electronic device to execute: in response to the page changes of the first application, the intelligent assistant obtains the first page data after the first application is updated.

[0073] According to the fifth aspect, or any implementation of the above fifth aspect, when the processor reads computer instructions from the memory, it also enables the electronic device to execute: if the user's third voice command is not received within a preset time, the intelligent assistant stops obtaining the first page data from the first application.

[0074] According to the fifth aspect, or any implementation of the fifth aspect, when the processor reads the computer instructions from the memory, the electronic device is further caused to execute: the intelligent assistant requests the window management system (WMS) to obtain the currently displayed first window based on the first voice command, where the first window is used to display the first application. Requesting the first application to obtain the first page data includes: the intelligent assistant obtaining the first page data through the interface of the first window.

[0075] In a sixth aspect, an electronic device is provided. The electronic device includes: a processor and a memory. The memory is coupled to the processor, and the memory is used to store computer program code. The computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes: an intelligent assistant in the electronic device receives a first voice command from the user, and the first voice command carries first information of a first application and second information of a second application. The first application is the application currently displayed by the electronic device. According to the first voice command, the intelligent assistant requests the first application to obtain first page data. According to the first page data, the intelligent assistant triggers the second application to execute a first event corresponding to the first voice command; wherein the first event is related to the first page data.

[0076] According to a sixth aspect, the first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via inter-process communication IPC.

[0077] According to the sixth aspect, or any implementation of the sixth aspect above, the intelligent assistant triggers the second application to execute the first event corresponding to the first voice command based on the first page data, including: the intelligent assistant instructs the second application to execute the first event in the background based on the first page data. Alternatively, the intelligent assistant triggers the electronic device to display the second application in response to the first voice command. The intelligent assistant instructs the second application to execute the first event based on the first page data.

[0078] According to the sixth aspect, or any implementation of the sixth aspect, the intelligent assistant triggers the second application to execute the first event based on the first page data, including: the intelligent assistant requests the second application to obtain the second page data. The intelligent assistant sends a second operation instruction to the second application based on the first page data and the second page data to trigger the second application to execute the first event.

[0079] According to the sixth aspect, or any implementation of the sixth aspect above, when the processor reads computer instructions from the memory, it also enables the electronic device to execute: in response to the page changes of the first application, the intelligent assistant obtains the first page data after the first application is updated.

[0080] According to the sixth aspect, or any implementation of the above sixth aspect, when the processor reads computer instructions from the memory, it also enables the electronic device to execute: if the user's second voice command is not received within a preset time, the intelligent assistant stops obtaining the first page data from the first application.

[0081] According to the sixth aspect, or any implementation of the sixth aspect, when the processor reads the computer instructions from the memory, the electronic device is further caused to execute: the intelligent assistant requests, based on the first voice command, from the window management system (WMS) to obtain a currently displayed first window, where the first window is used to display a first application. Requesting the first application to obtain first page data includes: the intelligent assistant obtaining the first page data through an interface of the first window.

[0082] In a seventh aspect, an electronic device is provided. The electronic device includes a processor and a memory. The memory is coupled to the processor and is configured to store computer program code. The computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the method according to the third aspect or any one of the embodiments of the third aspect.

[0083] In an eighth aspect, an electronic device is provided. The electronic device includes a processor and a memory. The memory is coupled to the processor and is configured to store computer program code. The computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device executes the method according to the fourth aspect or any one of the embodiments of the fourth aspect.

[0084] In a ninth aspect, an electronic device is provided, which has the function of implementing the method described in the first aspect and any possible implementation thereof; or, the electronic device has the function of implementing the method described in the second aspect and any possible implementation thereof; or, the electronic device has the function of implementing the method described in the third aspect and any possible implementation thereof; or, the electronic device has the function of implementing the method described in the fourth aspect and any possible implementation thereof. The function can be implemented by hardware, or by hardware executing corresponding software implementation. The hardware or software includes one or more modules corresponding to the above functions.

[0085] In a tenth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program (also referred to as instructions or code). When the computer program is executed by an electronic device, the electronic device performs the method of the first aspect or any one of the embodiments of the first aspect; or the electronic device performs the method of the second aspect or any one of the embodiments of the second aspect; or the electronic device performs the method of the third aspect or any one of the embodiments of the third aspect; or the electronic device performs the method of the fourth aspect or any one of the embodiments of the fourth aspect.

[0086] In the eleventh aspect, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes the method of the first aspect or any one of the embodiments of the first aspect; or, the electronic device executes the method of the second aspect or any one of the embodiments of the second aspect; or, the electronic device executes the method of the third aspect or any one of the embodiments of the third aspect; or, the electronic device executes the method of the fourth aspect or any one of the embodiments of the fourth aspect.

[0087] In the twelfth aspect, a circuit system is provided, which includes a processing circuit, and the processing circuit is configured to execute the method of the first aspect or any one of the embodiments of the first aspect; or, the processing circuit is configured to execute the method of the second aspect or any one of the embodiments of the second aspect; or, the processing circuit is configured to execute the method of the third aspect or any one of the embodiments of the third aspect; or, the processing circuit is configured to execute the method of the fourth aspect or any one of the embodiments of the fourth aspect.

[0088] In the thirteenth aspect, a chip system is provided, comprising at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor. When the at least one processor executes the instructions, the at least one processor executes the method of the first aspect or any one of the embodiments of the first aspect; or, the at least one processor executes the method of the second aspect or any one of the embodiments of the second aspect; or, the at least one processor executes the method of the third aspect or any one of the embodiments of the third aspect; or, the at least one processor executes the method of the fourth aspect or any one of the embodiments of the fourth aspect.

[0089] The technical effects of the aforementioned aspects can be referenced with each other and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 Schematic diagram of the accessibility service interaction scenario provided in the embodiment of the present application;

[0091] Figure 2 A schematic diagram of the interface provided in an embodiment of the present application;

[0092] Figure 3 A schematic diagram of a communication system to which the intelligent interaction method provided in an embodiment of the present application is applied;

[0093] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0094] Figure 5 A schematic diagram of the software structure block diagram of the electronic device provided in an embodiment of the present application;

[0095] Figure 6 Schematic diagram of the intelligent interaction method provided in this embodiment Figure 1 ;

[0096] Figure 7 A schematic diagram of a scenario for adding a schedule through an intelligent assistant provided in an embodiment of the present application;

[0097] Figure 8 Inter-module interaction diagram provided in this application embodiment Figure 1 ;

[0098] Figure 9 Inter-module interaction diagram provided in this application embodiment Figure 2 ;

[0099] Figure 10 Schematic diagram of adding notes through the smart assistant provided in the embodiment of this application Figure 1 ;

[0100] Figure 11 Schematic diagram of adding notes through the smart assistant provided in the embodiment of this application Figure 2 ;

[0101] Figure 12 Schematic diagram of the intelligent interaction method provided in this embodiment Figure 2 ;

[0102] Figure 13 Schematic diagram of the intelligent interaction method provided in this embodiment Figure 3 ;

[0103] Figure 14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two).

[0105] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0106] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0107] In some embodiments, the intelligent assistant can realize intelligent interaction of the user interface (UI) between the user and the electronic device. For example, the intelligent assistant can receive voice commands from the user to operate the electronic device, and the intelligent assistant can call the accessibility service based on the voice command to obtain the interface information of the current UI interface. In this way, the intelligent assistant can obtain the controls displayed on the UI interface indicated by the voice command based on the interface information, and trigger the application to execute the event corresponding to the control through the accessibility service. In some examples, the electronic device is, for example, a smart screen, a smart cockpit, a mobile phone and other devices. In some examples, the intelligent assistant is, for example, an intelligent assistant application installed in the electronic device, such as Xiaoyi.

[0108] In some examples, in response to user voice commands, such as Figure 1 As shown, the smart assistant requests the accessibility service to take a screenshot of the current UI interface. Afterwards, the smart assistant can obtain the screenshot and process the screenshot through image analysis technology to obtain interface information. In some examples, the interface information includes information about the controls displayed on the current display interface, such as position, size, etc. Figure 1 As shown, the smart assistant obtains the location information of the control instructed by the user's voice command based on the interface information, and sends the location information and operation method to the accessibility service. In this way, the accessibility service can instruct the application to perform the operation on the corresponding control.

[0109] For example, Figure 2 As shown, while the electronic device is displaying a chat interface, it receives a user voice command "Send XXX message." In response to this voice command, the intelligent assistant obtains a screenshot of the current display interface through the accessibility service and, through image analysis technology, obtains the location information of the input box 21 displayed on the current interface. The intelligent assistant can then send this location information and the input content indicated by the voice command to the accessibility service. In this way, the accessibility service can instruct the chat application to enter and display the required input content in the input box 21 and send the input content.

[0110] In other examples, in response to user voice commands, such as Figure 1 As shown, the smart assistant requests the accessibility service to obtain interface information for the currently displayed interface. The multiple controls displayed on the interface may form a control tree. A control tree refers to a structured representation used to manage and organize interface elements, such as controls, in graphical user interface programming. A control tree is typically presented in a tree-like structure, with the root node representing the entire application or window, and child nodes representing specific interface controls, such as buttons, text boxes, and lists. The accessibility service can use code to view all accessibility nodes, such as controls, in the current interface. For example, the accessibility service can use the code "AccessibilityNodeInfo rootNode = info.getRootInActivewindow()" to obtain information about the root node in the currently focused window, traverse the node tree using the code "traverseNode(rootNode)", output the node information using the code "private void traverseNode(AccessibilityNodeInfo node)", and traverse the child nodes using the code "traverseNode(childNode)". In this way, the accessibility service can output the interface information. Based on this interface information and the voice command, the smart assistant can instruct the application corresponding to the currently focused window to execute the corresponding voice command through the accessibility service.

[0111] It can be seen that the current intelligent interaction process relies on accessibility services. Accessibility services are primarily used to provide services for people with disabilities and do not fully match the functional implementation of intelligent assistants. Therefore, during intelligent interaction, the information provided by accessibility services to intelligent assistants cannot be customized, some unnecessary information is transmitted, which consumes power and memory, and some information required by intelligent assistants may not be provided by accessibility services.

[0112] It should be understood that inter-process communication (IPC) exists between smart assistants, accessibility services, and applications. Data transmission via IPC requires a long link, consumes more power, and is less efficient. Recognizing interface information through screenshots also involves high image recognition overhead.

[0113] Furthermore, for some private interfaces, current electronic devices do not allow screenshots to obtain private data, or screenshots will hide the private data. Therefore, smart assistants cannot provide users with a good intelligent interactive experience in such scenarios.

[0114] Thus, the embodiments of the present application provide an intelligent interaction method in which an intelligent assistant can directly interact with an application and, based on the acquired interface information, trigger the application to execute voice commands, thus simplifying the interaction process and improving communication efficiency. Furthermore, the direct interaction between the intelligent assistant and the application eliminates the need for redundant information exchange, reduces data transmission power consumption, and reduces the difficulty of developing the intelligent assistant.

[0115] Figure 3 Schematic diagram of a communication system for applying the intelligent interaction method provided in the embodiment of the present application. Figure 3 As shown, the communication system includes an electronic device 100 and a server 200.

[0116] In some embodiments, the electronic device 100 may be, for example, a smart screen, a smart cockpit, a mobile phone, a tablet computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a wearable device, an artificial intelligence (AI) device, or other terminal device. The operating system installed on the electronic device 100 includes but is not limited to This application does not limit the specific type of the electronic device 100 or the installed operating system.

[0117] Optionally, the server 200 may be a device or server with computing capabilities such as a cloud server or a network server. The server 200 may be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0118] In some embodiments, an intelligent assistant is installed in the electronic device 100, and the server 200 is an intelligent assistant server for supporting the cloud computing function of the intelligent assistant.

[0119] For example, Figure 4 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0120] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0121] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0122] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0123] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0124] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0125] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0126] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0127] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be manufactured using a liquid crystal display (LCD), for example, an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0128] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110. In some embodiments, the audio module 170 can include, for example, a speaker, a receiver, a microphone, a headphone jack, etc.

[0129] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the software structure of the electronic device 100 is exemplified by taking an operating system of a layered architecture as an example.

[0130] Figure 5 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0131] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, an operating system is divided into four layers: application layer, application framework layer, system library layer, and kernel layer, from top to bottom.

[0132] The application layer can include a series of application packages.

[0133] like Figure 5 As shown, the application package can include applications such as smart assistant, chat, calendar, notes, contacts, gallery, map, call, video, etc.

[0134] Among them, the intelligent assistant is used to realize intelligent interaction between the user and the electronic device. For example, the intelligent assistant can realize the interaction between the user and the UI interface of the electronic device 100 through voice, thereby realizing control of the electronic device 100.

[0135] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0136] like Figure 5 As shown, the application framework layer may include a window management system (WMS), a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0137] The window management system is used to manage window programs. The window management system can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0138] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0139] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0140] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0141] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0142] The Notification Manager allows applications to display notifications in the status bar. These can be used to convey informational messages and can disappear automatically after a short pause, without requiring user interaction. The Notification Manager can also display notifications in the system's top status bar as icons or scrolling text, such as notifications from background applications, or as dialog windows that appear on the screen.

[0143] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0144] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0145] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, etc.

[0146] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0147] A 2D graphics engine is a drawing engine for 2D drawings.

[0148] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0149] Figure 6 This is a flow chart of an intelligent interaction method provided by an embodiment of the present application. Figure 6 The specific order described below is for limitation only. It should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0150] S601: The intelligent assistant receives a user's voice command.

[0151] In some embodiments, the electronic device receives the user's voice through the voice system and sends the user's voice to the intelligent assistant. For example, the user's voice includes the user's voice command. For example, Figure 4 As shown, the electronic device collects user voice commands through the audio module and sends the voice commands to the smart assistant.

[0152] S602: The intelligent assistant sends a voice command recognition request to the server.

[0153] In some embodiments, in response to a received voice command, the intelligent assistant sends a voice command recognition request to the server, where the voice command recognition request is used to request the server to recognize the voice command received by the intelligent assistant. In some examples, the voice command recognition request carries the voice command.

[0154] Exemplarily, the intelligent assistant sends an automatic speech recognition (ASR) text command to the server, in which the voice command is carried, thereby triggering the server to convert the voice command into a text command for easy recognition and execution by the intelligent assistant.

[0155] S603: The server sends a voice command recognition response to the intelligent assistant.

[0156] In some embodiments, after receiving a voice command from an intelligent assistant, the server may process the voice command using a voice recognition algorithm to obtain a corresponding text command. The server may then feed the obtained text command back to the intelligent assistant. In some examples, the server sends a voice command recognition response to the intelligent assistant, and the voice command recognition response includes the text command.

[0157] Exemplarily, the voice command recognition response is, for example, a "Directives[]" statement, through which the server instructs the intelligent assistant on the text command that needs to be executed, and the text command corresponds to the user's voice command, thereby realizing the conversion of the voice command.

[0158] For example, if the user's voice command is "Order me some fried chicken," the Directives statement above consists of two parts: one for opening the food delivery app and the other for retrieving page data. This allows the smart assistant to open the food delivery app and, by retrieving the page data, place the fried chicken order.

[0159] In other embodiments, the electronic device supports an intelligent assistant to analyze voice commands on the terminal side. In this case, steps S602 and S603 are optional. After receiving the user's voice command, the intelligent assistant can directly analyze the voice command and obtain the corresponding voice recognition results.

[0160] In some other embodiments, the electronic device is configured with a related service based on speech-to-text (STT) technology, and the electronic device can also process the voice command through the related service and output the processing result to the intelligent assistant. For example, after receiving the voice command, the electronic device may not directly hand it over to the intelligent assistant for processing, but instead send the voice command to the related service based on STT technology for processing. Among them, STT technology is a technology that converts speech into written text.

[0161] S604: The intelligent assistant obtains the focus window and the application corresponding to the focus window.

[0162] The focus window refers to the interface element or application window that the current user operates or interacts with. The focus window is the window where the operating system receives user input, such as input by the user through voice, touch, keyboard, etc.

[0163] In some embodiments, the user's voice command is generally used to control the application displayed in the current focus window of the electronic device. Then, the intelligent assistant can determine the application displayed in the focus window based on the voice command recognition response received from the server and use the application as the focus application currently requiring interaction.

[0164] For example, the electronic device is Figure 2 In the scenario shown, the electronic device receives a user voice command "Send XX message" while displaying a chat app. The intelligent assistant interacts with the server, recognizes the response based on the received voice command, and determines that the focus app displayed in the focus window is the chat app.

[0165] In other embodiments, the user's voice command is also used to instruct the electronic device to launch a new application. Then, the intelligent assistant recognizes the response based on the received voice command, launches the application instructed by the user's voice command, and displays the application in the focus window, making it the focus application currently requiring interaction.

[0166] For example, if the user's voice command is "Order me some fried chicken," the intelligent assistant interacts with the server, recognizes the response based on the received voice command, and determines to launch a food delivery app. After the food delivery app is launched, the displayed food delivery app window becomes the focus window.

[0167] In some embodiments, a window management system (WMS) is a component of an operating system or a graphical user interface that manages and controls the display and interaction of multiple windows on the screen of an electronic device. For example, WMS can be used to implement functions such as window creation and destruction, focus management, event processing, window layout, and window switching. For example, WMS can be used to create new windows and destroy windows that are no longer needed. Each window can correspond to an ability instance, and window changes include changes in the ability instance. Windows include full-screen windows, pop-up windows, etc. For another example, WMS can be used to determine which window is the currently active window and receive user input (such as voice input events, etc.), that is, WMS can be used to determine the focus window.

[0168] In other examples, for other operating systems, each window may correspond to an activity instance, or to a UIviewcontroller instance.

[0169] For example, after receiving a user voice command, the smart assistant registers the focus window change event with the WMS. This allows the smart assistant to obtain the focus window API feedback from the WMS. Furthermore, after the focus window changes, the smart assistant can also obtain the updated focus window API feedback from the WMS. This enables subsequent interaction between the smart assistant and the focus application through the focus window API.

[0170] In some embodiments, after receiving a user voice command, or after receiving a voice command recognition response, the intelligent assistant can create and display a modal window, which can intuitively display the voice interaction process to the user.

[0171] For example, Figure 7 As shown in (a), when the electronic device displays the chat interface, it receives the user's voice command "Remember the chat information on the screen and add my speech to the calendar". The intelligent assistant sends the voice command to the server. In response to the voice command recognition response received from the server, Figure 7 As shown in (b), the intelligent assistant creates and displays a modal window 71, and displays the recognized voice command on the modal window 71, so that the user can determine the current voice command recognition status of the electronic device through the content displayed in the modal window 71.

[0172] In some examples, electronic devices display a modal window on the top of the current display interface through pop-up windows or other methods for easy viewing by users. Among them, the electronic device displays a modal window, but the display window of the modal window is not the focus window. The focus window is still the display window of the application corresponding to the user's voice command. Figure 7 In the scenario shown, the focus window is still the chat application window. This facilitates subsequent interaction between the smart assistant and the focus application displayed in the focus window.

[0173] S605: The intelligent assistant sends a page data acquisition request to the application.

[0174] S606: The application sends a page data acquisition response to the intelligent assistant.

[0175] In some embodiments, the user's intention when inputting a voice command is to interact with the UI interface of the focus application, such as the displayed content in the operation interface, such as text, pictures, etc. Then, after determining the focus application, the intelligent assistant can send a page data acquisition request to the focus application. The page data acquisition request is used to request to obtain page data of the current display interface to determine the display content with which the user intends to interact.

[0176] Correspondingly, after receiving the page data acquisition request sent by the smart assistant, the focus application can feedback the page data of the current display interface to the smart assistant.

[0177] In some examples, the focus application can be the application that provides the focus window. The focus application can change based on user operations. For example, if the electronic device displays the chat window of a chat application, the chat application becomes the focus application; if the electronic device switches from displaying the chat application's chat interface to displaying the shopping application's shopping interface in response to a user switching operation, the focus application will also become the shopping application.

[0178] The focus application can receive user voice commands, and the page data includes page data of the interface displayed in the focus window. In other examples, the focus application is an application launched by an intelligent assistant in response to a user voice command, and the page data includes page data of the interface displayed in the focus window after the application is launched and displayed in the focus window.

[0179] In some examples, during the graphical user interface editing process, the structured representation of the interface elements is managed and organized through a control tree. The control tree organizes controls through parent-child relationships to form a hierarchical structure. For example, a window control can have multiple child controls, such as a menu bar, toolbar, etc. Each control in the control tree can register an event detector. When the user interacts with the control, the event will propagate along the tree structure from the child control to the parent control, thereby realizing the response to the event. Then, the page data requested by the intelligent assistant to the application includes control tree data, which includes, for example, the controls included in the current display interface, control identifiers, the hierarchical relationship of each control, the display position of the control, the display size of the control, and other data. Subsequently, the intelligent assistant can determine the control indicated by the user's voice command based on the page data, and the control can also respond to the operation event of the voice command.

[0180] In some embodiments, the control tree can dynamically add, delete, or modify controls to support dynamic interface updates. For example, during the display of the focus application, events such as page content switching and page animation start / end may occur. In this case, the intelligent assistant can obtain the latest page data through interaction with the focus application. For example, the intelligent assistant registers page data update events with the focus application. In response to the page data update, the focus application can feedback the updated page data to the intelligent assistant.

[0181] In some embodiments, during the design of the user graphical interface of the application, developers can use native controls to build the user graphical interface, or use graphics provided by some third-party frameworks to build the user graphical interface. Among them, some third-party frameworks include, for example, Flutter, Qt, Weex or other frameworks. Flutter and Qt frameworks are used to build user interfaces and render graphics, each with a unique rendering implementation and architectural design. For example, Flutter uses the Skia graphics library as the underlying rendering engine. Skia is an open source 2D graphics library that supports multiple platforms, such as and desktop. During the application interface display process, Flutter draws directly to the screen through Skia, rather than relying on the platform's native controls. Flutter's user interface is composed of a tree-structured widget. For example, Qt provides Qt Quick, a high-performance graphical user interface framework based on the Qt meta-object language (QML) specifically for building user interfaces. Qt Quick can, for example, use OpenGL for hardware-accelerated rendering. Qt Widgets are C++-based user interface components.

[0182] In some examples, the user graphical interface is rendered through native controls and the above-mentioned example framework, and the control tree data corresponding to the user graphical interface includes the data of native controls corresponding to each node and / or widgets drawn by the third-party framework. Then, the focus application can obtain the data of each native control, but may not be able to directly obtain the data of the widget drawn by the third-party framework. In this way, the focus application can obtain a screenshot of the area where the widget drawn by the third-party framework is located by taking a screenshot. In some examples, if the display interface includes third-party controls, such as widgets drawn by a third-party framework, the page data fed back by the focus application to the smart assistant includes screenshot data corresponding to the display area of ​​the third-party control. In this way, after obtaining the page data, the smart assistant can also obtain information about each control on the current display interface by analyzing the image data of the screenshot data, thereby determining the content of the control indicated by the user's voice command.

[0183] S607: The intelligent assistant sends the page data to the server.

[0184] In some embodiments, the intelligent assistant obtains a response based on the received data and may obtain page data fed back by the application. The intelligent assistant may then send the page data to the server. In response, the server receives the page data sent by the intelligent assistant, triggering the server to analyze the page data based on the previously obtained voice command.

[0185] S608. The server sends a voice command execution instruction to the intelligent assistant.

[0186] In some embodiments, the server can determine the control that the user's voice instruction operates on based on the received page data and the text command previously obtained by analyzing the voice command, and the control corresponds to the currently displayed content of the focus application. The server can then generate a voice command execution instruction, which carries the identifier of the control that the voice command indicates the operation of and the corresponding control command. The server can then send the voice command execution instruction to the intelligent assistant. In response, the intelligent assistant can receive the voice command execution instruction.

[0187] For example, Figure 7 In the scenario shown, based on the received page data, the server can determine that the control being operated by the current user's voice command is text content 72 displayed on the interface, and the control command is to generate a corresponding schedule based on text content 72 and add the schedule to the calendar application. The server can then send a voice command execution instruction to the intelligent assistant, carrying an identifier of text content 72 and the corresponding control command.

[0188] In other embodiments, the electronic device supports an intelligent assistant to analyze page data on the terminal side. In this case, steps S607 and S608 are optional. After obtaining the page data, the intelligent assistant can directly analyze the page data in conjunction with the text command corresponding to the voice command and obtain the corresponding voice command execution instruction.

[0189] S609: The intelligent assistant sends an operation instruction to the application.

[0190] In some embodiments, after receiving the voice command execution instruction, the intelligent assistant obtains the control and application of the indicated operation and executes the corresponding control command. The application corresponding to the operation indicated by the voice command execution instruction can be the currently displayed focus application or another application.

[0191] Some examples, such as Figure 8 As shown, the smart assistant obtains the focus window of the voice command instruction and the focus application displayed in the focus window through the window management system (WMS). After that, the smart assistant obtains the control tree data sent by the focus application and generates the corresponding operation instruction, which is used to instruct the focus application to execute the user's voice command.

[0192] For example, Figure 2 As shown, the electronic device receives a user voice command "Send XX message" while displaying a chat application. In response to this voice command, the intelligent assistant determines that the focus application is the chat application, that the control indicated by the voice command is input box 21, and that the control command is to enter XX message in input box 21 and send it. The intelligent assistant then sends an operation instruction to the focus application, instructing the focus application to execute the control command.

[0193] In other examples, such as Figure 9 As shown, the smart assistant obtains the focus window supported by voice commands and the focus application displayed in the focus window, such as Application 1, through the window management system (WMS). The smart assistant then obtains the control tree data sent by the focus application and generates corresponding operation instructions. The operation instructions are used to instruct other applications, such as Application 2, to execute the user's voice command. The smart assistant can execute part of the voice command, while other applications execute other parts of the voice command.

[0194] Optionally, other applications can execute the user's voice commands in the background. Alternatively, other applications execute the user's voice commands in the foreground, that is, the electronic device switches other applications to the focus application. Optionally, as shown in step S604 above, the smart assistant has registered the focus window change event with the WMS. Then, after the focus window changes, the smart assistant can obtain the updated focus window API feedback from the WMS. In this way, the smart assistant can establish a connection with the new focus application through the updated focus window API, thereby obtaining the page data of the new focus application, so that the new focus can execute operation instructions.

[0195] For example, Figure 7 As shown in (b), in response to the user's voice command, the intelligent assistant obtains the text content 72 displayed on the interface of the control indicated by the user's voice command, and the control command is to generate a corresponding schedule based on the text content 72 and add the schedule to the calendar application. Then, the intelligent assistant can obtain the text content 72 based on the obtained page data, and determine the text content required to add the schedule in the text content 72, such as the meeting subject, meeting time, etc. Afterwards, the intelligent assistant can instruct to start the calendar application. For example, the intelligent assistant can send a calendar application startup request to the desktop to start the calendar application through the desktop. After the calendar application is started, the intelligent assistant can send an operation instruction to the calendar application, and carry the obtained text content required to add the schedule in the operation instruction. Then, in response to the operation instruction, such as Figure 7 As shown in (c), the calendar application can add a schedule on the corresponding date. The calendar application can be started in the background, that is, the user is not visible during the schedule addition process. Later, after the schedule is added, the smart assistant can obtain the schedule addition result. Figure 7 As shown in (b), the intelligent assistant can display the schedule adding result in the modal window 71, such as the calendar has been added and the specific schedule content added, so as to help the user understand the execution result of the voice command.

[0196] For example, Figure 10As shown, in the process of displaying a chat application, the electronic device obtains the user's voice command "take notes" through the voice system. After the smart assistant obtains the voice command, it can obtain the current focus application, such as if the focus application is a chat application. Then, the smart assistant can establish a connection with the chat application and obtain the control tree data provided by the chat application. Based on the voice command, the smart assistant determines that the user instructs to add the display content of the chat application, such as the chat content, in the note application. Then, the smart assistant can trigger the start of the note application to add the note content. Among them, after obtaining the chat content, the smart assistant needs to determine the insertion position of the chat content in the note application. Then, after the note application is started, the electronic device displays the note application in the focus window, the smart assistant establishes a connection with the note application, and obtains the control tree data provided by the note application to determine the insertion position of the chat content in the note application, and sends corresponding operation instructions to the note application. For example, as Figure 11 As shown in (a), the intelligent assistant obtains the chat content 111. Figure 11 As shown in (b), the note application is, for example, a memo application, and the focus window displays the memo application homepage. Then, the intelligent assistant obtains the identifier of the add control 112, and determines that the control command is to create a new memo, and adds the memo content as chat content 111. Afterwards, the intelligent assistant can send a corresponding operation instruction to the memo application to instruct the memo application to operate the add control 112 to create a new memo, and add the memo content as chat content 111 in the newly created memo. Figure 11 As shown in (c), after adding the memo content, the memo application can switch the corresponding display page and send the switched page data to the smart assistant so that the smart assistant can determine the execution status of the operation instruction. Figure 11 A modal window corresponding to the smart assistant can also be displayed in (a), (b) or (c), which is not shown in the figure.

[0197] In some embodiments, the intelligent assistant may receive a new voice command from the user within a preset time range after receiving the above voice command. In this case, the intelligent assistant may repeatedly execute steps S601-S609 to respond to the user's voice command. The preset time range may be, for example, 20 seconds, 30 seconds, etc.

[0198] In some embodiments, if the intelligent assistant does not receive any more user voice commands within a preset time, it may automatically exit. Alternatively, the intelligent assistant may exit after detecting a user-indicated exit operation, such as a voice command indicating exit. After exiting, the intelligent assistant stops receiving window change events and page data update events.

[0199] This allows the smart assistant to interact directly with the app, obtaining page data that is more suitable for voice command execution and avoiding the acquisition of redundant information. Furthermore, the smart interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption, and improving the efficiency of smart interaction.

[0200] Figure 12 This is a flow chart of another intelligent interaction method provided by the embodiment of the present application. Figure 12 The specific order described below is for limitation only. It should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0201] S1201: An intelligent assistant in an electronic device receives a first voice command from a user. The first voice command carries information about a first application, which is an application currently displayed on the electronic device.

[0202] For example, the first application is the focus application, and the user can control the first application through the first voice command. In this way, when it is inconvenient for the user to directly operate the electronic device display screen with his hands, the first application can also be controlled through the first voice command.

[0203] In some embodiments, the information of the first application carried in the first voice command may be information directly indicating the first application, or may be information associated with the first application.

[0204] For example, the first voice command is "Help me order fried chicken using a food delivery app." This first voice command directly includes the name of the food delivery app. Another example is the first voice command is "Help me order fried chicken." This first voice command does not directly include the name of the food delivery app, but only food delivery apps can implement the corresponding meal reservation function. That is, the first voice command carries relevant information about the food delivery app.

[0205] Optionally, for other contents of step S1201 , reference may be made to the relevant contents of steps S601 to S603 in the above embodiment.

[0206] S1202. The intelligent assistant requests the first application to obtain the first page data according to the first voice command.

[0207] In some embodiments, the intelligent assistant sends the received first voice command to a server, which analyzes the first voice command to obtain the first application that the first voice command indicates to be controlled. Alternatively, the intelligent assistant analyzes the first voice command on the client side to obtain the first application that the first voice command indicates to be controlled.

[0208] In some examples, after determining the first application, the intelligent assistant may request the first application to obtain the page data of the first application. For example, the intelligent assistant sends a first page data acquisition request to the first application based on the first voice command. In response to the received first page data acquisition request, the first application sends the first page data to the intelligent assistant. The first page data is, for example, the page data currently being displayed by the first application.

[0209] In some examples, the first page data includes control tree data of the first page, wherein the control tree data is transmitted between the first application and the intelligent assistant via IPC communication.

[0210] In this way, the intelligent assistant can directly realize IPC communication with the first application without going through the accessibility service, thereby improving the efficiency of intelligent interaction.

[0211] In some embodiments, a user generally controls the currently displayed first application through a voice command, or instructs the user to start the first application, thereby controlling the first application. Then, after receiving the first voice command, the intelligent assistant needs to determine the currently displayed first application and the interaction method with the first application.

[0212] In some examples, the electronic device is configured with a WMS, which is used to manage and control the display and interaction of multiple windows on the screen of the electronic device. Then, based on the first voice command, the intelligent assistant requests the WMS to obtain the currently displayed first window, which is used to display the first application. The first window is, for example, the focus window. The WMS can feedback the interface of the first window to the intelligent assistant, and the intelligent assistant obtains the first page data through the interface of the first window.

[0213] In some examples, the application that the user wants to control through the first voice command is the focused application, rather than other non-focused applications. In this case, when the electronic device displays multiple applications, the multiple applications are displayed in their own windows, including only one focused window, and the other windows are non-focused windows. The smart assistant can determine the focused window through the WMS, that is, determine the first application displayed in the focused window that the user currently wants to control by voice.

[0214] Optionally, for other contents of step S1202 , reference may be made to the relevant contents of steps S604 to S606 in the above embodiment.

[0215] S1203. The intelligent assistant triggers the first application to execute a first event corresponding to the first voice command based on the acquired first page data.

[0216] In some embodiments, after obtaining the first page data, the intelligent assistant can combine the analysis results of the first voice command to determine the first event that the first voice command indicates the first application to execute, thereby triggering the first application to execute the first event.

[0217] In some examples, the intelligent assistant triggers the first application to execute the first event corresponding to the first voice command based on the acquired first page data, including: the intelligent assistant obtains, based on the acquired first page data, a first control in the first application that is instructed to be operated by the first voice command. The intelligent assistant sends a first operation instruction to the first application, where the first operation instruction triggers the first application to operate the first control to execute the first event corresponding to the first voice command.

[0218] For example, Figure 2 As shown, when the electronic device is displaying a chat application (such as the first application), the intelligent assistant receives the user's first voice command "Send XX message". In response to the first voice command, the intelligent assistant requests the chat application to obtain the first page data. Based on the first page data, the intelligent assistant can determine that the first control indicated by the first voice command is the input box 21, and the control command is to enter the XX message in the input box 21 and send it. Then, the intelligent assistant can send an operation instruction to the chat application, which is used to instruct the chat application to execute the control command. Among them, the control command is, for example, a command for implementing the first event.

[0219] This allows the smart assistant to interact directly with the app, obtaining page data that is more suitable for voice command execution and avoiding the acquisition of redundant information. Furthermore, the smart interaction process does not rely on accessibility services, avoiding unnecessary memory and power consumption, and improving the efficiency of smart interaction.

[0220] In some embodiments, the electronic device displays a page of a second application. The intelligent assistant receives a second voice command from the user. The second voice command carries information about a third application. The intelligent assistant retrieves second page data of the second application based on the second voice command. Based on the second page data, the intelligent assistant triggers the third application to execute a second event.

[0221] In some examples, the intelligent assistant obtaining second page data of the second application based on the second voice command includes: the intelligent assistant sending a second page data obtaining request to the second application based on the second voice command. In response to the received second page data obtaining request, the second application sends the currently displayed second page data to the intelligent assistant.

[0222] In this way, the smart assistant can obtain the second page data of the second application through IPC communication with the second application currently being displayed.

[0223] In some examples, before the third application is triggered to execute the second event, the third application is in any of the following situations: not started, started but running in the background, started but not displayed in the focus window.

[0224] For example, the second voice command is "remember the chat information on the screen and add my speech to the corresponding schedule in the calendar application". The second voice command directly carries information of the calendar application (such as the third application).

[0225] For another example, the second voice command is "Remember the chat information on the screen and add my speech to the calendar." It can be seen that the second voice command carries relevant information about the third application, but the calendar application can be determined based on "calendar", that is, the third application.

[0226] In this way, when multiple applications are involved in the voice command, the intelligent assistant can also realize direct IPC communication with multiple applications, triggering corresponding application execution events, thereby improving communication efficiency.

[0227] In some examples, the intelligent assistant triggers the third application to execute the second event based on the second page data, including: the intelligent assistant instructs the third application to execute the second event in the background based on the second page data.

[0228] For example, the intelligent assistant sends a first execution command to the third application based on the second page data. In response to the received first execution command, the third application executes the second event in the background.

[0229] For example, Figure 7 As shown in (b), in response to the user's voice command, the intelligent assistant obtains the text content 72 displayed on the interface of the control indicated by the user's voice command, and the control command is to generate a corresponding schedule based on the text content 72 and add the schedule to the calendar application. Then, the intelligent assistant can obtain the text content 72 based on the obtained page data, and determine the text content required to add the schedule in the text content 72, such as the meeting subject, meeting time, etc. Afterwards, the intelligent assistant can instruct the background to start the calendar application. For example, the intelligent assistant can send a calendar application start request to the desktop to start the calendar application through the desktop. After the calendar application is started, the intelligent assistant can send an operation instruction (such as a first execution command) to the calendar application, and carry the obtained text content required to add the schedule in the operation instruction. Then, in response to the operation instruction, such as Figure 7 As shown in (c), the calendar application can add an event on the corresponding date. The calendar application can be started in the background, that is, the user is not visible during the event addition process.

[0230] In this way, the intelligent assistant obtains the page data of the currently displayed application according to the voice command, and triggers other applications to execute corresponding events in the background based on the page data, thereby realizing a highly efficient intelligent interaction process through direct interaction between applications.

[0231] In other examples, the intelligent assistant triggers a third application to execute a second event based on the second page data, including: the intelligent assistant triggers the electronic device to display the third application in response to the second voice command. The intelligent assistant instructs the third application to execute a second event based on the second page data, where the second event is related to the second page data.

[0232] For example, the intelligent assistant sends a second execution command to the third application based on the second page data. In response to the received second execution command, the third application executes the second event.

[0233] In some examples, the intelligent assistant requests the third application to obtain the third page data. The intelligent assistant sends a second operation instruction to the third application based on the second page data and the third page data to trigger the third application to execute the second event.

[0234] For example, the smart assistant sends a request to retrieve the third page data to a third application. In response to the request, the third application sends the currently displayed third page data to the smart assistant. Subsequently, the smart assistant sends a second operation instruction to the third application based on the second and third page data. In response to the second operation instruction, the third application executes a second event.

[0235] For example, Figure 10 As shown, the smart assistant determines, based on the voice command, that the user instructs to add the display content of the chat application, such as the chat content, in the note application. Then, the smart assistant can trigger the start of the note application to add the note content. After obtaining the chat content (such as the second page data), the smart assistant needs to determine the insertion position of the chat content in the note application. Then, after the note application is started, the electronic device displays the note application, and the window displaying the note application is the focus window. Afterwards, the smart assistant can obtain the control tree data (such as the third page data) provided by the note application to determine the insertion position of the chat content in the note application, and send corresponding operation instructions to the note application. For example, Figure 11 As shown in (a), the intelligent assistant obtains the chat content 111. Figure 11As shown in (b), the note application is, for example, a memo application, and the focus window displays the memo application homepage. Then, the intelligent assistant obtains the identifier of the add control 112, and determines that the control command is to create a new memo, and adds the memo content as chat content 111. Afterwards, the intelligent assistant can send a corresponding operation instruction to the memo application to instruct the memo application to operate the add control 112 to create a new memo, and add the memo content as chat content 111 in the newly created memo. Figure 11 As shown in (c), after adding the memo content, the memo application can switch the corresponding display page.

[0236] In this way, the intelligent assistant obtains the page data of the currently displayed application based on the voice command, and triggers other applications to run in the foreground based on this page data to obtain the page data of other applications. In this way, the intelligent assistant combines the page data obtained multiple times to trigger the execution event of the corresponding application. In this way, through direct interaction between applications, a highly efficient intelligent interaction process is achieved.

[0237] In some embodiments, in response to a page change of the first application, the intelligent assistant obtains updated first page data sent by the first application.

[0238] In this way, the intelligent assistant can determine the execution status of the first voice command based on the latest page display of the first application, or continue to execute the first voice command. Alternatively, after receiving other voice commands, it can also execute the corresponding voice command based on the updated first page data.

[0239] In some embodiments, if the user's third voice command is not received within a preset time, the intelligent assistant stops obtaining the first page data from the first application.

[0240] For example, if the smart assistant does not receive any more user voice commands within a preset time, it may automatically exit.

[0241] In this way, the smart assistant can automatically stop the intelligent interaction process, avoiding long waits for user voice commands and increasing device power consumption.

[0242] Optionally, for other contents of step S1203 , reference may be made to the relevant contents of steps S607 to S609 in the above embodiment.

[0243] In some scenarios, the first voice command may also directly carry information about the first application and information about another application that needs to be controlled.

[0244] Figure 13 This is a flow chart of another intelligent interaction method provided by the embodiment of the present application. Figure 13The specific order described below is for limitation only. It should be understood that in other embodiments, the order of some steps in the method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0245] S1301: An intelligent assistant in an electronic device receives a first voice command from a user. The first voice command carries first information of a first application and second information of a second application. The first application is an application currently displayed on the electronic device.

[0246] The first information of the first application carried in the first voice command may be information directly indicating the first application or information associated with the first application. The second information of the second application carried in the first voice command may be information directly indicating the second application or information associated with the second application.

[0247] In this way, the user can control multiple applications through the first voice command. Alternatively, the user can control other applications based on the page data of the currently displayed application.

[0248] S1302. The intelligent assistant requests the first application to obtain the first page data according to the first voice command.

[0249] In some examples, the first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via IPC communication.

[0250] In this way, the intelligent assistant can directly realize IPC communication with the first application without going through the accessibility service, thereby improving the efficiency of intelligent interaction.

[0251] S1303. The intelligent assistant triggers the second application to execute a first event corresponding to the first voice command based on the acquired first page data, where the first event is related to the first page data.

[0252] In some embodiments, the intelligent assistant triggers the second application to execute the first event corresponding to the first voice command based on the first page data, including: the intelligent assistant instructs the second application to execute the first event in the background based on the first page data.

[0253] For example, the intelligent assistant triggers the second application to execute the first event corresponding to the first voice command based on the acquired first page data, including: the intelligent assistant sends the first execution command to the second application based on the first page data. In response to the received first execution command, the second application executes the first event in the background.

[0254] In this way, the intelligent assistant obtains the page data of the currently displayed application according to the voice command, and triggers other applications to execute corresponding events in the background based on the page data, thereby realizing a highly efficient intelligent interaction process through direct interaction between applications.

[0255] In other embodiments, the intelligent assistant triggers the electronic device to display the second application in response to the first voice command. The intelligent assistant instructs the second application to execute the first event based on the first page data.

[0256] For example, the smart assistant triggers the electronic device to display the second application in response to the first voice command. The smart assistant sends a second execution command to the second application based on the first page data. In response to the received second execution command, the second application executes the first event.

[0257] In some examples, the intelligent assistant triggers the second application to execute the first event based on the first page data, including: the intelligent assistant requests the second application to obtain the second page data. The intelligent assistant sends a second operation instruction to the second application based on the first page data and the second page data to trigger the second application to execute the first event.

[0258] For example, the smart assistant sends a request to obtain the second page data to the second application. In response to the received request, the second application sends the currently displayed second page data to the smart assistant. Based on the first and second page data, the smart assistant sends a second operation instruction to the second application. In response to the second operation instruction, the second application executes the first event.

[0259] In this way, the intelligent assistant obtains the page data of the currently displayed application based on the voice command, and triggers other applications to run in the foreground based on this page data to obtain the page data of other applications. In this way, the intelligent assistant combines the page data obtained multiple times to trigger the execution event of the corresponding application. In this way, through direct interaction between applications, a highly efficient intelligent interaction process is achieved.

[0260] Optionally, for other contents of step S1301 to step S1303, reference may be made to the relevant contents of step S1201 to step S1203 in the above embodiment.

[0261] In some schemes, multiple embodiments of the present application can be combined and the combined scheme can be implemented. Optionally, some operations in the process of each method embodiment are optionally combined, and / or the order of some operations is optionally changed. In addition, the execution order between the steps of each process is only exemplary and does not constitute a limitation on the execution order between the steps. There can also be other execution orders between the steps. It is not intended to indicate that the execution order is the only order in which these operations can be performed. Ordinary technicians in this field will think of many ways to reorder the operations described herein. In addition, it should be noted that the process details involved in a certain embodiment of this article are also applicable to other embodiments in a similar manner, or different embodiments can be used in combination.

[0262] Furthermore, some steps in the method embodiments may be equivalently replaced with other possible steps. Alternatively, some steps in the method embodiments may be optional and may be deleted in certain usage scenarios. Alternatively, other possible steps may be added to the method embodiments.

[0263] Furthermore, the various method embodiments may be implemented separately or in combination.

[0264] Combination of the above Figure 6-Figure 13 The display intelligent interaction method provided by the embodiment of the present application is described in detail. Figure 14 The electronic device provided by the embodiments of the present application is described in detail.

[0265] In one possible design, Figure 14 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 14 As shown, the electronic device 1400 may include a transceiver unit 1401 and a processing unit 1402. The electronic device 1400 may be used to implement the functions of the electronic device involved in the above method embodiments.

[0266] Optionally, the transceiver unit 1401 is used to support the electronic device 1400 to execute Figure 6 S601, S602, S603, S607, and S608; and / or, for supporting the electronic device 1400 to execute Figure 12 S1201 in; and / or, for supporting the electronic device 1400 to execute Figure 13 S1301 in.

[0267] Optionally, the processing unit 1402 is configured to support the electronic device 1400 in executing Figure 6 S604, S605, S606 and S609 in; and / or, for supporting the electronic device 1400 to execute Figure 12 S1202 and S1203 in; and / or, for supporting the electronic device 1400 to execute Figure 13 S1302 and S1303 in.

[0268] Optionally, the transceiver unit 1401 may include a receiving unit and a transmitting unit, and may be implemented by a transceiver or a transceiver-related circuit component, and may be a transceiver or a transceiver module. The operations and / or functions of each unit in the electronic device 1400 are respectively to implement the corresponding process of the display intelligent interaction method described in the above method embodiment. All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional unit. For the sake of brevity, they are not repeated here.

[0269] Optionally, Figure 14 The electronic device 1400 shown may further include a storage unit ( Figure 14 (not shown), the storage unit stores a program or instruction. When the transceiver unit 1401 and the processing unit 1402 execute the program or instruction, Figure 14 The electronic device 1400 shown can execute the display intelligent interaction method described in the above method embodiment.

[0270] Figure 14 The technical effects of the electronic device 1400 shown can refer to the technical effects of the display intelligent interaction method described in the above method embodiment, and will not be repeated here.

[0271] In addition to being in the form of the electronic device 1400, the technical solution provided in this application may also be a functional unit or chip in the electronic device, or a device used in conjunction with the electronic device.

[0272] An embodiment of the present application also provides a chip system, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the chip system implements the method in any of the above method embodiments.

[0273] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0274] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0275] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0276] It should be understood that each step in the above method embodiment can be completed by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0277] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer, the computer executes the above-mentioned related steps to implement the intelligent interaction method in the above-mentioned embodiment.

[0278] An embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the intelligent interaction method in the above-mentioned embodiment.

[0279] In addition, embodiments of the present application further provide a device. Specifically, the device may be a component or module, and may include one or more processors and a memory connected to each other. The memory is used to store a computer program. When the computer program is executed by one or more processors, the device performs the intelligent interaction method described in each of the above method embodiments.

[0280] The apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0281] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner or can be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC).

[0282] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed; that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-mentioned systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0283] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of modules or units, which can be electrical, mechanical or other forms.

[0284] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0285] Computer-readable storage media include, but are not limited to, any of the following: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media that can store program code.

[0286] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An intelligent interaction method, characterized in that: The method comprises: An intelligent assistant in an electronic device receives a first voice command from a user, where the first voice command carries information about a first application, and the first application is an application currently displayed on the electronic device; The intelligent assistant requests the first application to obtain the first page data according to the first voice command; The intelligent assistant triggers the first application to execute a first event corresponding to the first voice command based on the acquired first page data.

2. The method according to claim 1, characterized in that The first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via inter-process communication (IPC).

3. The method according to claim 1 or 2, characterized in that The intelligent assistant triggers the first application to execute a first event corresponding to the first voice command based on the acquired first page data, including: The intelligent assistant obtains, based on the obtained first page data, a first control in the first application that is instructed to be operated by the first voice command; The intelligent assistant sends a first operation instruction to the first application, and the first operation instruction triggers the first application to operate the first control to execute the first event corresponding to the first voice command.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The electronic device displays a page of a second application; The intelligent assistant receives a second voice command from the user; wherein the second voice command carries information about the third application; The intelligent assistant obtains the second page data of the second application according to the second voice command; The intelligent assistant triggers the third application to execute a second event based on the second page data, where the second event is related to the second page data.

5. The method according to claim 4, characterized in that The intelligent assistant triggers the third application to execute a second event according to the second page data, including: The intelligent assistant instructs the third application to execute the second event in the background according to the second page data; or, In response to the second voice command, the intelligent assistant triggers the electronic device to display the third application; The intelligent assistant instructs the third application to execute the second event based on the second page data.

6. The method according to claim 5, characterized in that The intelligent assistant triggers the third application to execute the second event according to the second page data, including: The intelligent assistant requests the third application to obtain the third page data; The intelligent assistant sends a second operation instruction to the third application based on the second page data and the third page data to trigger the third application to execute the second event.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: In response to a page change of the first application, the intelligent assistant obtains updated first page data of the first application.

8. The method according to any one of claims 1 to 7, characterized in that Before the intelligent assistant requests the first application to obtain the first page data according to the first voice command, the method further includes: The intelligent assistant requests the window management system WMS to obtain a currently displayed first window according to the first voice command, where the first window is used to display the first application; The requesting the first application to obtain the first page data includes: The intelligent assistant obtains the first page data through the interface of the first window.

9. An intelligent interaction method, characterized in that: The method comprises: An intelligent assistant in an electronic device receives a first voice command from a user, where the first voice command carries first information of a first application and second information of a second application, where the first application is an application currently displayed on the electronic device; The intelligent assistant requests the first application to obtain the first page data according to the first voice command; The intelligent assistant triggers the second application to execute a first event corresponding to the first voice command based on the first page data; wherein the first event is related to the first page data.

10. The method according to claim 9, characterized in that The first page data includes control tree data of the first page; wherein the control tree data is transmitted between the first application and the intelligent assistant via inter-process communication (IPC).

11. The method according to claim 9 or 10, characterized in that The intelligent assistant triggers the second application to execute a first event corresponding to the first voice command according to the first page data, including: The intelligent assistant instructs the second application to execute the first event in the background according to the first page data; or, In response to the first voice command, the intelligent assistant triggers the electronic device to display the second application; The intelligent assistant instructs the second application to execute the first event based on the first page data.

12. The method according to claim 11, characterized in that The intelligent assistant triggers the second application to execute the first event according to the first page data, including: The intelligent assistant requests the second application to obtain the second page data; The intelligent assistant sends a second operation instruction to the second application based on the first page data and the second page data to trigger the second application to execute the first event.

13. The method according to any one of claims 9 to 12, characterized in that: The method further comprises: In response to a page change of the first application, the intelligent assistant obtains updated first page data of the first application.

14. The method according to any one of claims 9 to 13, characterized in that: Before the intelligent assistant requests the first application to obtain the first page data according to the first voice command, the method further includes: The intelligent assistant requests the window management system WMS to obtain a currently displayed first window according to the first voice command, where the first window is used to display the first application; The requesting the first application to obtain the first page data includes: The intelligent assistant obtains the first page data through the interface of the first window.

15. An electronic device, characterized in that: include: A processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor reads the computer instructions from the memory, the electronic device executes the method according to any one of claims 1 to 8; or, the electronic device executes the method according to any one of claims 9 to 14.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a computer program, which, when executed on an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 8; or enables the electronic device to execute the method according to any one of claims 9 to 14.

17. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 8; or the computer is enabled to execute the method according to any one of claims 9 to 14.

Images