Voice analysis method, electronic device, readable storage medium and chip system

Abstract

The application is suitable for the terminal technical field, and provides a voice analysis method, an electronic device, a readable storage medium and a chip system.The method comprises the following steps: acquiring a voice instruction and information sent by a running application program, the voice instruction being used for instructing a terminal device to perform an operation, and the information sent by the application program comprising reminding information used for reminding a user; and determining a user intention corresponding to the voice instruction according to the voice instruction and the reminding information.Through acquiring the information sent by the application program, the information sent by the application program is used as a factor for determining the user intention, the accuracy of determining the user intention corresponding to the voice instruction can be improved, and thus the efficiency of voice interaction between the terminal device and the user can be improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a voice parsing method, electronic device, readable storage medium, and chip system. Background Technology

[0002] With the continuous development of terminal devices, they can not only perform actions based on user-triggered clicks and other operations, but also detect user voices through voice assistants and perform actions based on those voices.

[0003] In existing technologies, terminal devices can detect voice commands issued by users through voice assistants, and parse the voice commands issued by users in combination with the interface content currently displayed on the terminal device's interface to determine the user's intent corresponding to the voice command. Then, the terminal device can be controlled to perform operations that match the user's intent.

[0004] However, in some scenarios, terminal devices may fail to accurately understand user intent, potentially triggering incorrect operations or requiring repeated questioning of the user, resulting in low efficiency in user interaction. Summary of the Invention

[0005] This application provides a voice parsing method, an electronic device, a readable storage medium, and a chip system, which solves the problem of low efficiency in user interaction of terminal devices in certain scenarios in the prior art.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, a speech parsing method is provided, including:

[0008] The system acquires voice commands and information issued by running applications, wherein the voice commands are used to instruct the terminal device to perform operations, and the information issued by the application includes reminders for reminding the user.

[0009] Based on the voice command and the reminder information, determine the user intent corresponding to the voice command.

[0010] By acquiring information from the running application when a voice command is received, and using this information as a factor in determining the user's intent, the accuracy of determining the user's intent corresponding to the voice command can be improved, thereby increasing the efficiency of voice interaction between the terminal device and the user.

[0011] In a first possible implementation of the first aspect, before determining the user intent corresponding to the voice command based on the voice command and the reminder information, the method further includes:

[0012] Obtain a first application list and a second application list, wherein the first application list is a list of all applications installed on the terminal device, and the second application list is a list of applications currently running on the terminal device;

[0013] Based on the first application list and the second application list, determine the identifier of the application corresponding to the voice command and the running status of the application;

[0014] The step of determining the user intent corresponding to the voice command based on the voice command and the reminder information includes:

[0015] If the application is running in the background, the user intent corresponding to the voice command is determined based on the reminder information, the voice command, and the application's identifier.

[0016] If the application is running in the foreground, then based on the current interface of the application, the interface information corresponding to the current interface is obtained, and based on the voice command, the reminder information and the interface information, the user intent corresponding to the voice command is determined.

[0017] Based on the first application list and the second application list, the application corresponding to the voice command is determined, thereby determining the running status of the application. Then, based on different running statuses, different methods can be used to determine the user intent corresponding to the voice command, which can improve the flexibility of determining the user intent.

[0018] If the application is running in the foreground, its interface information can be obtained. This allows the user's intent to be determined based on the voice commands and reminders, combined with the obtained interface information, thus improving the accuracy of determining the user's intent.

[0019] Based on the first possible implementation of the first aspect, in the second possible implementation of the first aspect, obtaining the interface information corresponding to the current interface of the application includes:

[0020] Extract the current interface to obtain the interface content included in the current interface;

[0021] The interface content is parsed to obtain the interface information corresponding to the application.

[0022] Based on any of the possible implementations of the first aspect described above, in a third possible implementation of the first aspect, the acquisition of voice commands and information issued by the running application includes:

[0023] The voice command is acquired at the first moment;

[0024] Based on the first moment, obtain the information emitted by each of the applications that were running within a preset time period before the first moment.

[0025] By acquiring voice commands at the first moment and then acquiring information sent by the application within a preset time before the first moment, the workload of acquiring information sent by the application can be reduced, and the diversity and flexibility of acquiring voice commands and information sent by the application can be improved.

[0026] Based on any of the possible implementations of the first aspect other than the third one, in the fourth possible implementation of the first aspect, obtaining the information emitted by the running application includes:

[0027] Get information sent by the running application in real time.

[0028] By acquiring information from the application in real time, the user's intent corresponding to the voice command can be determined promptly by combining the acquired information. This improves the efficiency of determining user intent and enhances the diversity and flexibility of acquiring information from the application.

[0029] Based on any of the possible implementations of the first aspect described above, in the fifth possible implementation of the first aspect, the step of obtaining the information emitted by the running application includes:

[0030] The audio data played by the terminal device is obtained through a pre-set interface;

[0031] The audio data is converted using Automatic Speech Recognition (ASR) technology to obtain the text information sent by the application.

[0032] By acquiring audio data from the application and converting it into text-based information, the flexibility of acquiring information from the application can be improved.

[0033] Based on any of the possible implementations of the first aspect other than the fifth one, in the sixth possible implementation of the first aspect, obtaining the information emitted by the running application includes:

[0034] The text data sent by the application is extracted through a pre-set interface to obtain the information in text form sent by the application.

[0035] By extracting text data to obtain information sent by the application, the efficiency and flexibility of obtaining such information can be improved.

[0036] Based on any of the above possible implementations of the first aspect, in the seventh possible implementation of the first aspect, before determining the user intent corresponding to the voice command based on the voice command and the reminder information, the method further includes:

[0037] The voice commands are converted using ASR technology to obtain text commands in text form;

[0038] The step of determining the user intent corresponding to the voice command based on the voice command and the reminder information includes:

[0039] Based on the text instruction and the reminder information, determine the user intent corresponding to the voice instruction.

[0040] By converting voice commands into text commands, and then using these text commands in conjunction with reminder information to determine user intent, the flexibility and diversity of determining user intent can be improved.

[0041] Based on the seventh possible implementation of the first aspect, in the eighth possible implementation of the first aspect, the step of using ASR technology to convert the voice command into a text command includes:

[0042] The voice command is denoised using voice enhancement technology to obtain the denoised voice command.

[0043] The ASR technology is used to convert the denoised speech command into a text command.

[0044] By denoising the voice commands and then converting the denoised voice commands into text commands, the accuracy of converting text commands can be improved, thereby improving the accuracy of determining the user's intent.

[0045] Based on any of the above possible implementations of the first aspect, in the ninth possible implementation of the first aspect, before acquiring the voice command and the information issued by the running application, the method further includes:

[0046] Based on multiple types of sample data, various correlations are established between different types of sample data. The multiple types of sample data include: sample reminder information, sample interface content, sample voice commands, and sample user intents. The multiple correlations include: the correlation between the sample user intents and the sample reminder information, the correlation between the sample user intents and the sample voice commands, and the correlation between the sample user intents and the sample interface content.

[0047] Training is performed based on various relationships to obtain a fusion model, which is a single model or a group of models.

[0048] Based on the ninth possible implementation of the first aspect, in the tenth possible implementation of the first aspect, determining the user intent corresponding to the voice command based on the voice command and the reminder information includes:

[0049] The user intent corresponding to the voice command is determined by combining the voice command and the reminder information through the fusion model.

[0050] By parsing the acquired voice commands and reminder information using a fusion model, the user intent output by the fusion model that matches the voice commands and reminder information can be obtained, which can improve the accuracy of determining the user intent.

[0051] Based on any of the above possible implementations of the first aspect, in the eleventh possible implementation of the first aspect, after determining the user intent corresponding to the voice command based on the voice command and the reminder information, the method further includes:

[0052] Based on the user intent, the intent execution interface is invoked to perform an operation that matches the user intent.

[0053] Based on any of the above possible implementations of the first aspect, in the twelfth possible implementation of the first aspect, the method is applied in a multi-device scenario, the multi-device scenario including a first terminal device and a second terminal device, the first terminal device being connected to the second terminal device;

[0054] The information obtained from acquiring voice commands and the information emitted by the running application includes:

[0055] The first terminal device acquires voice commands and information issued by the application running on the first terminal device;

[0056] The first terminal device sends an information request instruction to the second terminal device according to the voice command. The information request instruction is used to instruct the second terminal device to obtain and provide feedback to the first terminal device on the information sent by the application running on the second terminal device.

[0057] The first terminal device receives information from the running application fed back by the second terminal device.

[0058] In a multi-device scenario, any terminal device that collects voice commands can control other devices in the scenario based on the voice commands, which can improve the flexibility of voice command control of terminal devices.

[0059] Secondly, a speech parsing device is provided, comprising:

[0060] The first acquisition module is used to acquire voice commands and information issued by the running application. The voice commands are used to instruct the terminal device to perform operations, and the information issued by the application includes reminder information for reminding the user.

[0061] The first determining module is used to determine the user intent corresponding to the voice command based on the voice command and the reminder information.

[0062] In a first possible implementation of the second aspect, the device further includes:

[0063] The second acquisition module is used to acquire a first application list and a second application list, wherein the first application list is a list of applications installed on the terminal device, and the second application list is a list of applications currently running on the terminal device.

[0064] The second determining module is used to determine the identifier of the application corresponding to the voice command and the running status of the application based on the first application list and the second application list.

[0065] The first determining module is specifically configured to: if the application is running in the background, determine the user intent corresponding to the voice command based on the reminder information, the voice command, and the application's identifier; if the application is running in the foreground, obtain the interface information corresponding to the current interface based on the application's current interface, and determine the user intent corresponding to the voice command based on the voice command, the reminder information, and the interface information.

[0066] Based on the first possible implementation of the second aspect, in the second possible implementation of the second aspect, the first determining module is further specifically used to extract the current interface to obtain the interface content included in the current interface; and to parse the interface content to obtain the interface information corresponding to the application.

[0067] Based on any of the above possible implementations of the second aspect, in a third possible implementation of the second aspect, the first acquisition module is specifically used to acquire the voice command at a first moment; and according to the first moment, acquire information issued by each of the applications running within a preset time before the first moment.

[0068] Based on any of the possible implementations of the second aspect other than the third, in the fourth possible implementation of the second aspect, the first acquisition module is specifically used to acquire information issued by the running application in real time.

[0069] Based on any of the above possible implementations of the second aspect, in the fifth possible implementation of the second aspect, the first acquisition module is further specifically used to acquire the audio data broadcast by the terminal device through a pre-set interface; and to convert the audio data using automatic speech recognition technology (ASR) to obtain the text information sent by the application.

[0070] Based on any of the other possible implementations of the second aspect besides the fifth one, in the sixth possible implementation of the second aspect, the first acquisition module is further specifically used to extract the text data sent by the application through a pre-set interface to obtain the information in text form sent by the application.

[0071] Based on any of the possible implementations of the second aspect described above, in a seventh possible implementation of the second aspect, the apparatus further includes:

[0072] The conversion module is used to convert the voice commands using ASR technology to obtain text commands in text form;

[0073] The first determining module is further specifically used to determine the user intent corresponding to the voice command based on the text command and the reminder information.

[0074] Based on the seventh possible implementation of the second aspect, in the eighth possible implementation of the second aspect, the conversion module is specifically used to denoise the voice command using speech enhancement technology to obtain a denoised voice command; and to convert the denoised voice command using ASR technology to obtain the text command in text form.

[0075] Based on any of the possible implementations of the second aspect described above, in the ninth possible implementation of the second aspect, the apparatus further includes:

[0076] A module is established to establish multiple relationships between different types of sample data based on various sample data. The various sample data include: sample reminder information, sample interface content, sample voice commands, and sample user intents. The various relationships include: the relationship between the sample user intent and the sample reminder information, the relationship between the sample user intent and the sample voice commands, and the relationship between the sample user intent and the sample interface content.

[0077] The training module is used to train a fusion model based on various relationships to obtain a fusion model, which is a single model or a group of models.

[0078] Based on the ninth possible implementation of the second aspect, in the tenth possible implementation of the second aspect, the first determining module is further specifically used to determine the user intent corresponding to the voice command by combining the voice command and the reminder information through the fusion model.

[0079] Based on any of the possible implementations of the second aspect described above, in the eleventh possible implementation of the second aspect, the apparatus further includes:

[0080] The execution module is used to call the intent execution interface according to the user intent and perform an operation that matches the user intent.

[0081] Based on any of the above possible implementations of the second aspect, in the twelfth possible implementation of the second aspect, the device is applied in a multi-device scenario, the multi-device scenario includes a first terminal device and a second terminal device, the first terminal device and the second terminal device are connected.

[0082] The first acquisition module is further specifically used for the first terminal device to acquire voice commands and information issued by the application running on the first terminal device; and according to the voice commands, to send an information request command to the second terminal device, and then receive information issued by the running application from the second terminal device. The information request command is used to instruct the second terminal device to acquire and send back to the first terminal device the information issued by the application running on the second terminal device.

[0083] Thirdly, an electronic device is provided, comprising: a processor for running a computer program stored in a memory to implement the speech parsing method as described in any one of the first aspects above.

[0084] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the speech parsing method as described in any one of the first aspects above.

[0085] Fifthly, a chip system is provided, the chip system including a memory and a processor, the processor executing a computer program stored in the memory to implement the speech parsing method as described in any one of the first aspects above.

[0086] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0087] Figure 1A A schematic diagram of the interface of a shopping application provided in an embodiment of this application;

[0088] Figure 1B A schematic diagram of a system settings interface provided in an embodiment of this application;

[0089] Figure 1C A schematic diagram of the interface of a map application provided in an embodiment of this application;

[0090] Figure 2 A schematic diagram of a speech parsing scenario involved in a speech parsing method provided in an embodiment of this application;

[0091] Figure 3 A schematic flowchart illustrating a speech parsing method provided in an embodiment of this application;

[0092] Figure 4 A schematic diagram of a software architecture-based process for obtaining reminder information is provided for an embodiment of this application;

[0093] Figure 5 A schematic diagram of another software architecture-based process for obtaining reminder information provided in an embodiment of this application;

[0094] Figure 6 A schematic diagram illustrating a multi-device voice parsing process provided in an embodiment of this application;

[0095] Figure 7 A structural block diagram of a speech parsing device provided in an embodiment of this application;

[0096] Figure 8 A structural block diagram of another speech parsing device provided in an embodiment of this application;

[0097] Figure 9A structural block diagram of another speech parsing device provided in an embodiment of this application;

[0098] Figure 10 A structural block diagram of another speech parsing device provided in an embodiment of this application;

[0099] Figure 11 A structural block diagram of another speech parsing device provided in an embodiment of this application;

[0100] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0101] Figure 13 This is a software structure block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0102] In the following description, specific details such as particular system architectures and technologies are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known model training methods, speech parsing methods, interface parsing methods, and electronic devices are omitted to avoid unnecessary detail from obscuring the description of this application.

[0103] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “the,” “the,” and “the” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise.

[0104] First, we will introduce the scenarios in which terminal devices interact with users via voice.

[0105] During operation, the terminal device can activate its voice assistant. The terminal device can then collect and parse the user's voice commands to obtain the information contained within them and determine the user's intent. For example, the terminal device can use its voice assistant to turn air conditioning, audio systems, and lights on or off.

[0106] If the information included in the voice command is incomplete and the voice assistant cannot understand the user's intent, the voice assistant can determine the user's intent by combining it with the content displayed on the current screen. For example, see... Figure 1A If the terminal device's current interface is that of a shopping application, the voice assistant can perform shopping operations based on voice commands; see also Figure 1BIf the terminal device is currently displaying the system settings interface, the voice assistant can perform the operation of turning on Bluetooth based on voice commands.

[0107] Furthermore, if the voice assistant still cannot determine the user's intent after combining the content displayed on the current interface, the voice assistant will no longer control the terminal device to execute the operation corresponding to the voice command. Alternatively, the voice assistant may need to ask the user for more information in order to determine the user's intent corresponding to the voice command based on the user's response.

[0108] For example, see Figure 1C When a terminal device is using a map application for route navigation, the map application may remind the user that "a faster route has been found, which can get you to your destination 5 minutes earlier." If the voice assistant detects a user's voice command to "switch routes," but the voice assistant cannot determine the corresponding user intent based on the voice command, then the voice assistant will no longer control the terminal device to perform the route switching operation on the map application, or the voice assistant will need to ask the user again.

[0109] Alternatively, in a multi-device scenario, both induction cookers are operating, and one of them prompts the user, "It has been on high for 20 minutes; we recommend lowering the temperature and simmering on low." If the voice assistant detects that the user's voice command is "Adjust the temperature to 200 degrees," the voice assistant can control both induction cookers to adjust to 200 degrees, causing the devices to perform an erroneous operation.

[0110] Therefore, this application proposes a voice parsing method. The voice assistant obtains various information from the application and, based on the reminder information for the user among these information, accurately determines the user's intent corresponding to the voice command based on the user's voice command and, in conjunction with the interface content currently displayed, controls the terminal device to execute an operation matching the user's intent, thereby improving the efficiency of user interaction with the voice assistant.

[0111] See Figure 2 , Figure 2 This is a schematic diagram of a speech parsing scenario involved in a speech parsing method provided in an embodiment of this application. The speech parsing scenario may include at least one terminal device 210, wherein each terminal device 210 may be located in the same network, and each terminal device may include various types of devices such as smart TVs, smart speakers, routers, and projectors. This embodiment of the application does not limit the type of terminal device.

[0112] For each of the at least one terminal device 210, during operation, the terminal device 210 may activate a voice assistant, obtain voice commands issued by the user and various information issued by the currently running application through the voice assistant, and then determine the user intent corresponding to the voice command based on the voice command and the reminder information in the various information.

[0113] Specifically, when the voice assistant detects that the user has issued a voice command, the voice assistant can parse the voice command and obtain the information issued by the application within a preset time. Based on the reminder information issued to the user in the information, combined with the interface content displayed on the current interface of the terminal device, the voice assistant outputs the user intent corresponding to the voice command through a pre-trained fusion model. Thus, the voice assistant can control the terminal device 210 to perform an operation that matches the user intent.

[0114] For example, corresponding to the map navigation scenario described above, after detecting a user's voice command to "switch routes," the voice assistant can also obtain information from the map application and, based on the "find a faster route" prompt among various pieces of information, combined with the currently displayed interface of the map application, determine that the user's intent for the voice command is to switch the navigation route to the discovered faster route. Accordingly, the voice assistant can control the terminal device to switch navigation routes within the map application.

[0115] Similarly, in the multi-device scenario described above, after the voice assistant detects that the user has issued a voice command to "adjust the temperature to 200 degrees", it can obtain information from each induction cooker. Based on the "suggest lowering the temperature" reminder from one of the induction cookers, the voice assistant can determine that the user's intention for the voice command is to adjust the temperature of the induction cooker that issued the reminder to 200 degrees. Then, the voice assistant can control the induction cooker to lower the temperature to 200 degrees.

[0116] It should be noted that the above scenario describes a situation where the voice assistant first detects the user's voice command and then determines the user's intent based on the reminder information among various pieces of information acquired. In practical applications, the voice assistant can also acquire various pieces of information, including reminder information, issued by the application in real time, and then determine the user's intent by combining the reminder information among these various pieces of information when detecting the user's voice command. This application does not limit the order in which reminder information is acquired and voice command is acquired.

[0117] Furthermore, the voice commands involved in the embodiments of this application can be commands used to instruct the terminal device to perform operations. For example, the voice command is used to instruct the terminal device to purchase goods, turn a function on or off, or adjust the state of the terminal device. The embodiments of this application do not limit the operations that the voice commands instruct the terminal device to perform.

[0118] Furthermore, applications can emit various types of information during operation, including reminders for the user. Correspondingly, voice assistants can acquire this information through various methods. For example, a voice assistant can intercept information emitted by the application through a pre-set interface; it can also receive various types of information proactively sent to it by the application; and it can acquire information emitted by the application through other means. This application embodiment does not limit the types of information emitted by the application or the methods of acquiring such information.

[0119] Furthermore, the above examples of voice parsing scenarios involving multiple devices are for illustrative purposes only. In practical applications, voice parsing methods can be applied to a variety of different voice parsing scenarios, such as controlling smart home devices through voice assistants, controlling in-vehicle devices through voice assistants, and other scenarios where terminal devices are controlled through voice assistants. This application does not limit the voice parsing scenarios.

[0120] Figure 3 This is a schematic flowchart illustrating a speech parsing method provided in an embodiment of this application. It is intended as an example and not a limitation. This method can be applied to the aforementioned terminal device. See also... Figure 3 The method includes:

[0121] Step 301: Train the model based on multiple sample data to obtain a fusion model.

[0122] The fusion model can be a single model or a model group consisting of multiple models. The models in the model group can work together. This application embodiment does not limit the number of models in the fusion model. For example, if the fusion model is a model group, in the process of the fusion model determining the user intent, the first model in the model group can determine the domain to which the voice command belongs based on information such as the received voice command, reminder information, and interface content. Then, the voice command, reminder information, and interface content can be input into the model in the model group corresponding to that domain to obtain the user intent output by that model.

[0123] Terminal devices can interact with users via voice assistants. The voice assistant can parse the user's voice commands using a pre-trained fusion model to determine the user's intent, thereby controlling the terminal device to perform actions matching that intent. Therefore, before engaging in voice interaction with the user, the terminal device can train a fusion model based on various sample data.

[0124] The sample data can include various types of data such as sample reminder information, sample interface content, sample voice commands, and sample user intents. Sample reminder information can be messages from the application to the user; sample interface content can be the content displayed on each interface when the application is launched on the terminal device; sample voice commands can be the sounds emitted by the user to the terminal device; and sample user intents can be the different operations performed by the terminal device based on the voice commands, interface content, and reminder information.

[0125] In one optional implementation, the terminal device can establish associations between different types of sample data based on various sample data, thereby enabling it to input various sample data, such as sample reminder information, sample interface content, and sample voice commands, from these associations into a pre-set initial model to obtain the initial user intent output by the initial model. Subsequently, the terminal device can adjust the initial model based on the initial user intent output by the initial model, combined with the sample user intents corresponding to the multiple sample data input into the initial model from the various associations, and repeat the above process until the trained model can accurately output the sample user intents corresponding to the various sample data, such as sample reminder information, sample interface content, and sample voice commands.

[0126] Specifically, the terminal device can first establish a correlation between different types of sample data based on the sample data of each type and the detected user-triggered operations. In other words, the terminal device can establish a correlation between any sample user intent and a certain sample data of other types of sample data.

[0127] In the process of establishing associations, the terminal device can establish associations between each sample user intent and three other types of sample data. That is, for each sample user intent, it can establish an association between the sample user intent and at least one sample reminder message, an association between the sample user intent and at least one sample voice command, and an association between the sample user intent and at least one sample interface content, thereby obtaining associations between multiple types of sample data.

[0128] For example, the terminal device can associate the sample user intent "switch to a faster route" with the sample reminder information "a faster route has been found, which can bring you to your destination 5 minutes earlier", it can also associate the sample user intent "switch to a faster route" with the sample voice command "switch route", and it can also associate the sample user intent "switch to a faster route" with the interface corresponding to the map application in the sample interface content.

[0129] After establishing multiple associations based on a subset of sample data (e.g., 80%), the terminal device can use these associations as label data. It then inputs a large amount of sample data (such as sample reminders, sample interface content, and sample voice commands) into a pre-set initial model to obtain the initial user intent output by the model. Based on the associations used as label data, the device compares the sample user intent corresponding to the sample data with the initial user intent, thereby adjusting the parameters of the initial model. By training the initial model multiple times in this manner, enabling it to accurately output sample user intents corresponding to various sample data sets, the training of the initial model is complete, resulting in the fusion model.

[0130] In addition, after the terminal device trains the fusion model, it can use the remaining sample data (such as 20%) from the large sample data to test the fusion model and determine the similarity between the initial user intent output by the fusion model and the sample user intent. Based on the obtained similarity, it can be determined whether the fusion model needs to be trained further.

[0131] It should be noted that in practical applications, the fusion model can be trained not only through terminal devices based on a large amount of sample data, but also through servers based on sample data. This application does not limit the execution entity of the trained fusion model.

[0132] Furthermore, a large amount of sample data can be obtained manually, or it can be collected through other methods. This application does not limit the method of sample data collection. For example, the reminder information used to remind users in the application can be captured manually.

[0133] Furthermore, after the fusion model is trained, the terminal device can add the fusion model to a pre-set voice assistant so that the voice assistant can determine the user's intent through the fusion model; the terminal device can also set up an interface corresponding to the voice assistant so that the voice assistant can call the fusion model through the interface. The embodiments of this application do not limit the way in which the voice assistant and the fusion model work together.

[0134] Step 302: Obtain information sent by the currently running application.

[0135] During operation, the application can emit various types of information, including reminders. These reminders are used to alert the user. Furthermore, the reminders can be derived from speech broadcast by the application or from text messages provided by the application; this embodiment does not limit the type of reminder information.

[0136] During operation, the terminal device can launch different applications, each of which can send out various types of information, including reminders. The voice assistant can then obtain these various types of information from the applications so that, in subsequent steps, it can determine the user's intent based on the reminders among these various types of information and the user's voice commands.

[0137] Correspondingly, the fusion model trained in step 301 is trained based on a large number of sample reminder information, which is manually collected. Therefore, the fusion model can identify reminder information among the various information obtained by the voice assistant from the application, and thus output the user's intent based on the reminder information in subsequent steps. However, if the fusion model does not recognize other information besides reminder information, then when the voice assistant inputs other information from the various information into the fusion model, the fusion model cannot output the user's intent, but instead outputs an anomaly (such as the fusion model outputting nothing or something else).

[0138] In one optional embodiment, the terminal device can activate a voice assistant upon startup. The voice assistant can monitor various running applications and acquire various types of information emitted by these applications (e.g., intercepting various types of information emitted by applications). Accordingly, during operation, the applications can, based on the constantly changing scenarios of the terminal device, provide users with reminders corresponding to the changing scenarios through voice broadcasts. When the application broadcasts reminder information, the voice assistant can acquire the audio data of the reminder information through a pre-set interface, and then use automatic speech recognition (ASR) technology to convert the audio data into text to obtain the application's reminder information.

[0139] For example, see Figure 4In a navigation scenario using a map application, the map application at the APP layer can use the received traffic information and the current location of the terminal device to play the audio data "Find a faster route" through the media player in the system layer. The voice assistant can then obtain the audio data from the media player through a pre-set interface, convert the audio data, and obtain the corresponding text "Find a faster route", which is the reminder information from the map application.

[0140] In another alternative embodiment, similar to obtaining the reminder information via voice broadcast as described above, if the application does not have the voice broadcast function enabled or does not possess such a function, the application will remind the user by displaying text information. Correspondingly, when the application sends a reminder, the voice assistant can obtain the text-based reminder information from the application through a pre-set interface.

[0141] It should be noted that when the application plays audio data, the voice assistant can also obtain reminder information in other ways. For example, the voice assistant can obtain text information corresponding to the played audio data from the application through a pre-set interface, and then use the obtained text information as the application's reminder message.

[0142] Additionally, when the application detects that a reminder needs to be given to the user, it can proactively send the audio data or text information to be read aloud to the voice assistant, eliminating the need for the voice assistant to retrieve the audio data or text information again. That is, see [link to relevant documentation]. Figure 5 The application can send reminder information to the voice assistant at the APP layer, and the voice assistant can obtain the reminder information without going through the system layer player.

[0143] Furthermore, after sending audio data or text information to the voice assistant, the application can continue to play the audio data or display the text information to the user, or it can play the audio data or text information through the voice assistant. This application embodiment does not limit the method of playing reminder information.

[0144] It should also be noted that the terminal device can pre-train a recognition model for identifying reminder information based on a large number of manually collected sample reminder messages, combined with other information from the application. Subsequently, the terminal device can use this recognition model to identify various pieces of information obtained by the voice assistant, thereby obtaining the reminder information. Of course, in practical applications, the voice assistant can also determine the reminder information issued by the application through other methods; this application embodiment does not limit the method of obtaining reminder information.

[0145] Step 303: Obtain the voice command issued by the user and convert the voice command into a text command.

[0146] Voice commands are instructions used to direct terminal devices to perform operations.

[0147] After an application on a terminal device sends a notification, the user may issue a voice command based on the notification, instructing the terminal device or application to perform a related operation. Correspondingly, the terminal device can use a voice assistant to collect the user's voice commands and convert them into text commands.

[0148] Furthermore, after the terminal device is started, a pre-set voice assistant can be activated first, so that the voice assistant can continuously collect the voice commands issued by the user. This allows the terminal device to be controlled to perform corresponding operations based on the voice commands, thereby improving the efficiency of voice interaction between the terminal device and the user.

[0149] Specifically, after the voice assistant is activated on the terminal device, it can access the device's microphone to collect audio data and obtain the user's voice commands. Then, the voice assistant can use speech enhancement technology to filter out noise from the voice commands, resulting in filtered voice commands, which are the user's spoken voice. The voice assistant can then use Automatic Speech Reduction (ASR) to convert the filtered voice commands into text commands.

[0150] Furthermore, during the process of collecting voice commands, the voice assistant can use wake words to collect voice commands. The voice assistant can continuously collect audio data through the microphone, and when the collected audio data includes a wake word, the voice assistant can use the audio data within a preset time after the wake word as the voice command.

[0151] For example, if the wake word is "Xiao E" and the preset time is 10 seconds, and the audio data collected by the voice assistant is "Xiao E, switch routes", then after detecting the wake word "Xiao E", the voice assistant can use the audio data "switch routes" within 10 seconds after the wake word as the voice command, filter it through speech enhancement, and then perform text conversion through ASR technology to obtain the text command "switch routes".

[0152] In addition to wake words, voice assistants can also obtain voice commands by combining wake words with end words. After detecting a wake word, the voice assistant can treat all audio data following the wake word as voice commands until it detects an end word.

[0153] For example, if the wake word is "Xiao E" and the end word is "Goodbye", and the audio data collected by the voice assistant is "Xiao E, switch routes, goodbye", then after detecting the wake word "Xiao E", the voice assistant can continue to use the collected audio data "switch routes" as a voice command. When the voice assistant detects the end word "Goodbye", it will no longer use the collected audio data as a voice command.

[0154] It should be noted that the embodiments in this application are illustrated by executing step 302 first and then step 303. In practical applications, the terminal device may also execute step 303 first and then step 302, indicating that the voice assistant first collects the voice command issued by the user, and then obtains the information issued by the application within a preset time, thereby determining the user's intent corresponding to the voice command based on the reminder information issued by the application and the voice command.

[0155] Specifically, the voice assistant does not obtain information issued by the application in real time. Instead, it first collects the voice commands issued by the user and selects information issued by the application within a preset time before that time based on the time when the voice commands are collected, thereby obtaining the reminder information issued by the application.

[0156] For example, if the preset time is 2 minutes, and the voice assistant of the terminal device collects the user's voice command at 11:10, the terminal device can traverse the applications running from 11:08 to 11:10 and obtain the information issued by each application running between 11:08 and 11:10.

[0157] Of course, the terminal device may also execute only step 303 and not step 302. This application embodiment does not limit the order of steps 302 and 303, nor does it limit whether the terminal device executes step 302.

[0158] Specifically, even when the application doesn't provide any prompts, the user can still issue commands based on the current scenario, and the terminal device can then receive these voice commands. For example, in a navigation scenario using a map application, if the application doesn't issue a "find shorter route" notification, but the user encounters traffic congestion ahead or needs to change their destination, they can issue a voice command to the voice assistant to "switch route" or "change destination."

[0159] In addition, voice assistants can continuously obtain information about the current scene from the terminal device or application to constantly determine the user's current environment. When the voice assistant detects that the user's current environment does not match the state of the terminal device or application, it can remind the user and determine whether the state of the terminal device or application needs to be adjusted based on the user's voice commands.

[0160] For example, in a navigation scenario using a map application, if the application detects traffic congestion on the planned route, it can send this congestion information as a notification to the voice assistant. The voice assistant can then ask the user, "There's traffic congestion on road segment A. Do you want to change the navigation route?" If the voice assistant detects the user's answer, "Change route," it can instruct the map application to switch the navigation route.

[0161] Step 304: Based on the text command corresponding to the voice command, determine the application corresponding to the text command.

[0162] After receiving a text command, the terminal device can locate the application corresponding to that command. Since applications running in the foreground can send notifications, and applications running in the background may also send notifications, the terminal device can first determine the application corresponding to the text command based on the voice command. This allows the terminal device to retrieve the interface of the determined application in subsequent steps, or to determine the user's intent based on the text command and the notification information.

[0163] In one optional embodiment, the terminal device can obtain a first application list and a second application list, wherein the first application list is a list of all applications installed on the terminal device, and the second application list is a list of all applications currently running on the terminal device, including applications running in the foreground and applications running in the background.

[0164] Then, the terminal device can search for the application associated with the text command from the first application list and the second application list, in conjunction with the text command corresponding to the voice command.

[0165] The terminal device can search for an application that matches the text command from the first application list and the second application list based on the text command corresponding to the voice command, thereby determining the running status of the matching application. The running status indicates whether the terminal device is running the application, and whether the terminal device is running the application in the foreground or in the background.

[0166] If the application corresponding to the running status indicator and the text command is an application running in the foreground of the terminal device, the terminal device can execute step 305 to obtain the interface information of the application. If the application corresponding to the running status indicator and the text command is an application running in the background of the terminal device, the terminal device can obtain the identifier of the application and skip step 305 to execute step 306. Combining the determined application identifier, the user intent corresponding to the voice command is determined through a fusion model.

[0167] For example, if the text command is "Change the destination to location A", the application associated with the text command can be an application with navigation function; if the text command is "Reduce the volume to 20", the application associated with the text command can be an application with audio and video data playback function; if the text command is "Open application B", the application associated with the text command can be an application with the same or similar name as the application in the text command.

[0168] Step 305: If the application corresponding to the text instruction is an application running in the foreground of the terminal device, then obtain the interface information based on the current interface content of the terminal device.

[0169] The current interface refers to the interface currently displayed on the terminal device. The interface content can include all the elements displayed on the current interface, such as text, images, and controls. The interface content can also include information such as the text position of the text, the image position of the image, and the category and position of the controls.

[0170] Furthermore, the interface information is determined based on the interface content and is used to represent the current scene of the terminal device and the actions associated with the current scene (such as the actions that the terminal device is currently performing, and / or the actions that the terminal device may perform).

[0171] For example, if the current interface is that of a map application, the interface information corresponding to the current interface may include: the current scenario of the terminal device is a map navigation scenario, the current action of the terminal device is navigating to the destination, and the possible actions that the terminal device may perform include: changing the route, changing the destination, and stopping navigation.

[0172] In one optional embodiment, the terminal device can obtain the current interface of the terminal device, that is, the interface corresponding to the application, through system services, identify each element in the interface content, and then parse the identified elements to obtain the interface information of the application.

[0173] Step 306: Determine the user intent corresponding to the voice command based on the information issued by the application and the text command corresponding to the voice command.

[0174] The user intent can be obtained from the output of the fusion model trained in step 301. For example, if the terminal device does not have the application's interface information, the voice assistant of the terminal device can input the information issued by the application, the text command corresponding to the voice command, and the identifier corresponding to the application into the fusion model to obtain the user intent output by the fusion model; if the terminal device has the application's interface information, the voice assistant can input the information issued by the application, the text command corresponding to the voice command, and the interface information corresponding to the application into the fusion model to obtain the user intent output by the fusion model.

[0175] In one optional embodiment, if the application corresponding to the voice command is a foreground application, after obtaining the application's interface information, the terminal device can simultaneously input the text command, interface information, and information issued by the application into the fusion model. The fusion model parses the text command, interface information, and information issued by the application and outputs an operation that matches the text command, interface information, and information issued by the application, which is the user intent corresponding to the voice command. This allows the terminal device to execute the matching operation based on the user intent in subsequent steps.

[0176] For example, if the fusion model consists of a group of models, the terminal device can first input each piece of information into the first model in the model group, analyze each piece of information through the first model, output the domain to which the voice command belongs, then determine the model that matches the domain to which the voice command belongs from the multiple models in the model group, and input information such as text commands, interface information and information issued by the application into the model, and parse each piece of information through the model to obtain the user intent corresponding to the voice command.

[0177] In another optional embodiment, if the application corresponding to the voice command is a background application, the terminal device skips step 305 after completing step 304, and inputs the text command, the information issued by the application, and the application identifier obtained in step 304 into the fusion model to obtain the user intent output by the fusion model.

[0178] It should be noted that when the terminal device inputs information sent by the application into the fusion model, if the information sent by the application is obtained by the terminal device after executing step 302 and then step 303, the terminal device can select the information sent by the application within a preset time period to input into the fusion model according to the time when the voice command is obtained; if the information sent by the application is obtained by the terminal device after executing step 303 and then step 302, the terminal device can input all the information sent by the application within the preset time period into the fusion model.

[0179] Of course, the terminal device can also use other methods to select information from the input fusion model, and this application embodiment does not limit this.

[0180] Step 307: Perform an operation that matches the user's intent.

[0181] After determining the user's intent corresponding to the voice command, the terminal device can call the pre-set intent execution interface to control the terminal device to perform the operation that matches the user's intent, respond to the voice command issued by the user, and complete the voice interaction between the terminal device and the user, thereby reducing the process and steps required for the terminal device to conduct voice interaction with the user.

[0182] It should be noted that in practical applications, step 301 is an optional step. The terminal device can execute step 301 only once. That is, after the fusion model is trained, it is not necessary to execute step 301 again to train the fusion model each time the user intent corresponding to the voice command is determined. Moreover, step 305 is also an optional step. If the terminal device determines in step 304 that the application corresponding to the voice command is an application running in the foreground of the terminal device, then the terminal device can execute step 305; if the terminal device determines in step 304 that the application corresponding to the voice command is an application running in the background of the terminal device, then the terminal device can skip step 305 and execute step 306.

[0183] Furthermore, the above embodiments only illustrate a single terminal device. In practical applications, the above voice parsing method can also be applied to scenarios with multiple devices. For example, it can be applied to scenarios where different smart home devices are controlled through a terminal device, or to scenarios where in-vehicle devices are controlled through a terminal device, or to other scenarios involving multiple devices. This application embodiment does not limit the scope of the application.

[0184] See Figure 6 , Figure 6 A flowchart illustrating the process of voice parsing across multiple devices is shown. Figure 6 Taking the first terminal device and the second terminal device as examples, this paper illustrates a method for any terminal device to access the network for voice parsing. This method may include:

[0185] Step 601: When the first terminal device accesses the network, it broadcasts a first application list to other devices in the network and requests a second application list from the second terminal device.

[0186] The first application list can be a list of all applications currently running on the first terminal device. Similarly, the second application list can be a list of all applications currently running on the second terminal device.

[0187] Furthermore, the network accessed by the first terminal device can be a local area network (LAN), a wide area network (WAN), or the Internet. For example, in a smart home scenario, the first terminal device can access a LAN; in a distributed application scenario, the first terminal device can access a WAN. This application embodiment does not limit this.

[0188] When the first terminal device detects network access, it can generate a list request message and obtain the first application list of the first terminal device. It then broadcasts the first application list and the list request message to other terminal devices in the network so that other terminal devices in the network can receive the first application list and send back the corresponding application list to the first terminal device according to the list request message.

[0189] It should be noted that during operation, the first terminal device can continuously open and close applications, thus the first application list of the first terminal device is constantly changing. Accordingly, when the first terminal device detects a newly opened application or an application being closed, it can update the first application list and broadcast the updated first application list to other devices in the network.

[0190] Step 602: The second terminal device receives the first application list from the first terminal device and sends the second application list of the second terminal device back to the first terminal device.

[0191] Step 603: When the first terminal device detects a voice command issued by the user, it converts the voice command into a text command and obtains the reminder information and interface information of the first terminal device.

[0192] Step 604: The first terminal device sends an information request instruction to the second terminal device.

[0193] The information request instruction is used to request reminder information and interface information from the second terminal device. Furthermore, this embodiment illustrates the example of executing step 603 first and then step 604. However, in practical applications, the first terminal device can execute steps 603 and 604 simultaneously; that is, while converting the voice command, the first terminal device can also send an information request instruction to the second terminal device.

[0194] Of course, the first terminal device can also execute step 604 first and then step 603. This application embodiment does not limit the order in which the first terminal device executes steps 603 and 604. However, in the process of the first terminal device executing step 604 first and then step 603, the first terminal device does not obtain the reminder information and interface information of the second terminal device based on the detected voice command. Instead, it periodically obtains the reminder information and interface information of the second terminal device, adopting a process similar to steps 302 to 303 above, and pre-obtains the reminder information and interface information of the second terminal device. When a voice command issued by the user is detected, the user intent corresponding to the voice command can be determined in subsequent steps.

[0195] Step 605: The second terminal device obtains the reminder information and interface information of the second terminal device according to the information request instruction sent by the first terminal device, and sends the reminder information and interface information of the second terminal device back to the first terminal device.

[0196] The processes of obtaining reminder information and interface information in steps 603 and 605 are similar to those in steps 302 and 305, which involve obtaining information sent by the application and obtaining interface information, and will not be described in detail here.

[0197] Step 606: The first terminal device determines the application corresponding to the voice command and the user intent corresponding to the voice command based on the multiple reminder messages obtained and the text command.

[0198] Corresponding to steps 602 and 605, if the first terminal device obtains the application's interface information in steps 602 and 605, then during the execution of step 606, the first terminal device can also determine the user's intent based on the obtained interface information. However, if the first terminal device does not obtain the application's interface information in steps 602 and 605, then during the execution of step 606, the first terminal device can determine the user's intent based on the prompt information and text instructions, combined with the determined application identifier. This will not be elaborated further here.

[0199] Step 607: If the application corresponding to the voice command is an application running on the second terminal device, then the first terminal device sends the user intent to the second terminal device.

[0200] If the first terminal device determines that the user's voice command is directed to a specific application on the second terminal device, the first terminal device can send the user's identified and parsed intent to the second terminal device. This allows the second terminal device to invoke the intent execution interface based on the user's intent, control the second terminal device to perform the operation corresponding to the user's intent, and achieve multi-device collaborative work.

[0201] Step 608: The second terminal device executes the operation corresponding to the received user intent.

[0202] Step 609: If the application corresponding to the voice command is an application running on the first terminal device, then the first terminal device performs an operation corresponding to the user's intent.

[0203] It should be noted that the process of obtaining interface information and reminder information in steps 603, 605, 606, 608 and 609, determining the user's intent based on the interface information and reminder information and the text instructions, and then performing the corresponding operation according to the user's intent is similar to the process in steps 302 to 306, and will not be described again here.

[0204] In addition, this application embodiment takes the voice assistant of a terminal device as an example. Some applications installed on the terminal device can also interact with the user by voice. The installed applications can also use the above-mentioned voice parsing method to determine the user's intention based on the voice command and realize the control of the terminal device or each application.

[0205] For example, if it is inconvenient for users to operate map applications while driving, they can issue voice commands to the map applications through voice interaction. The map applications can then switch navigation routes, change destinations, control in-vehicle equipment, or control the vehicle based on the voice commands.

[0206] In summary, the voice parsing method provided in this application improves the accuracy of determining the user intent corresponding to the voice command by acquiring the information issued by the running application when a voice command is obtained, and also using the information issued by the application as a factor in determining the user intent. This can improve the efficiency of voice interaction between the terminal device and the user.

[0207] Upon receiving a voice command from a user, the system can convert the voice command into a text command and acquire information from the application. A pre-trained fusion model then parses the text command and the application's information to output the user's intent corresponding to the voice command. By acquiring and incorporating the application's information as a factor in determining the user's intent, the accuracy of identifying the user's intent corresponding to the voice command can be improved, thereby increasing the efficiency of voice interaction between the terminal device and the user.

[0208] Furthermore, when a user command corresponds to an application running in the foreground of the terminal device, the terminal device can also obtain the application's interface information. Based on the text command and the information issued by the application, the fusion model, combined with the application's interface information, can more accurately determine the user's intent corresponding to the voice command, thereby improving the accuracy of determining the user's intent.

[0209] In addition, in multi-device scenarios, any terminal device that collects voice commands can control other devices in the multi-device scenario based on the voice commands, which can improve the flexibility of voice command control of terminal devices.

[0210] In addition, applications with voice interaction capabilities can determine the user's intent corresponding to the voice command by combining the prompt information and interface information, which can improve the breadth and flexibility of the application in parsing voice commands and obtaining user intent.

[0211] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0212] Corresponding to the speech parsing method described in the above embodiments, Figure 7 This is a structural block diagram of a speech parsing device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0213] Reference Figure 7 The device includes:

[0214] The first acquisition module 701 is used to acquire voice commands and information issued by the running application. The voice commands are used to instruct the terminal device to perform operations, and the information issued by the application includes reminder information for reminding the user.

[0215] The first determining module 702 is used to determine the user intent corresponding to the voice command based on the voice command and the reminder information.

[0216] Optional, see Figure 8 The device also includes:

[0217] The second acquisition module 703 is used to acquire a first application list and a second application list, wherein the first application list is a list of applications installed on the terminal device, and the second application list is a list of applications currently running on the terminal device.

[0218] The second determining module 704 is used to determine the identifier of the application corresponding to the voice command and the running status of the application based on the first application list and the second application list.

[0219] The first determining module 702 is specifically used to determine the user intent corresponding to the voice command based on the reminder information, the voice command, and the identifier of the application if the application is running in the background; and to obtain the interface information corresponding to the current interface based on the current interface of the application, and to determine the user intent corresponding to the voice command based on the voice command, the reminder information, and the interface information.

[0220] Optionally, the first determining module 702 is further configured to extract the current interface to obtain the interface content included in the current interface; and to parse the interface content to obtain the interface information corresponding to the application.

[0221] Optionally, the first acquisition module 701 is specifically used to acquire the voice command at a first moment; and based on the first moment, acquire information issued by each application running within a preset time before the first moment.

[0222] Optionally, the first acquisition module 701 is specifically used to acquire information emitted by the running application in real time.

[0223] Optionally, the first acquisition module 701 is further configured to acquire audio data broadcast by the terminal device through a pre-set interface; and to convert the audio data using automatic speech recognition (ASR) technology to obtain text information issued by the application.

[0224] Optionally, the first acquisition module 701 is also specifically used to extract the text data sent by the application through a pre-set interface to obtain the information in text form sent by the application.

[0225] Optional, see Figure 9 The device also includes:

[0226] The conversion module 705 is used to convert the voice command using ASR technology to obtain a text command in text form.

[0227] The first determining module 702 is further specifically used to determine the user intent corresponding to the voice command based on the text command and the reminder information.

[0228] Optionally, the conversion module 705 is specifically used to denoise the voice command using speech enhancement technology to obtain a denoised voice command; and to convert the denoised voice command using ASR technology to obtain a text command in text form.

[0229] Optional, see Figure 10 The device also includes:

[0230] The module 706 is used to establish multiple relationships between different types of sample data based on various sample data. The various sample data include: sample reminder information, sample interface content, sample voice commands, and sample user intents. The various relationships include: the relationship between the sample user intent and the sample reminder information, the relationship between the sample user intent and the sample voice command, and the relationship between the sample user intent and the sample interface content.

[0231] Training module 707 is used to train a fusion model based on multiple such relationships, which can be a single model or a group of models.

[0232] Optionally, the first determining module 702 is further configured to determine the user intent corresponding to the voice command by combining the voice command and the reminder information through the fusion model.

[0233] Optional, see Figure 11 The device also includes:

[0234] The execution module 708 is used to call the intent execution interface according to the user intent and perform the operation that matches the user intent.

[0235] Optionally, the device is applied in a multi-device scenario, which includes a first terminal device and a second terminal device, and the first terminal device is connected to the second terminal device.

[0236] The first acquisition module 701 is further specifically used for the first terminal device to acquire voice commands and information issued by the application running on the first terminal device; according to the voice command, to send an information request command to the second terminal device, the information request command being used to instruct the second terminal device to acquire and feed back to the first terminal device the information issued by the application running on the second terminal device; and to receive the information issued by the running application fed back by the second terminal device.

[0237] In summary, the voice parsing device provided in this application improves the accuracy of determining the user intent corresponding to the voice command by acquiring information issued by the running application when a voice command is received, and also using the information issued by the application as a factor in determining the user intent. This improves the efficiency of voice interaction between the terminal device and the user.

[0238] The following description uses a terminal device as an example to illustrate the electronic equipment involved in the embodiments of this application. Please refer to... Figure 12 , Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0239] The electronic device may include a processor 1210, an external memory interface 1220, an internal memory 1221, a universal serial bus (USB) interface 1230, a charging management module 1240, a power management module 1241, a battery 1242, an antenna 1, an antenna 2, a mobile communication module 1250, a wireless communication module 1260, an audio module 1270, a speaker 1270A, a receiver 1270B, a microphone 1270C, a headphone jack 1270D, a sensor module 1280, buttons 1290, a motor 1291, an indicator 1292, a camera 1293, a display screen 1294, and a subscriber identification module (SIM) card interface 1295, etc. The sensor module 1280 may include a pressure sensor 1280A, a gyroscope sensor 1280B, a barometric pressure sensor 1280C, a magnetic sensor 1280D, an accelerometer sensor 1280E, a distance sensor 1280F, a proximity sensor 1280G, a fingerprint sensor 1280H, a temperature sensor 1280J, a touch sensor 1280K, an ambient light sensor 1280L, a bone conduction sensor 1280M, etc.

[0240] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0241] Processor 1210 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0242] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0243] The processor 1210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1210 is a cache memory. This memory can store instructions or data that the processor 1210 has just used or that are used repeatedly. If the processor 1210 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1210, and thus improves the efficiency of the system.

[0244] In some embodiments, the processor 1210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0245] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 1210 may include multiple I2C buses. The processor 1210 can couple to the touch sensor 1280K, charger, flash, camera 1293, etc., through different I2C bus interfaces. For example, the processor 1210 can couple to the touch sensor 1280K through the I2C interface, enabling the processor 1210 and the touch sensor 1280K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device.

[0246] The I2S interface can be used for audio communication. In some embodiments, the processor 1210 may include multiple I2S buses. The processor 1210 can be coupled to the audio module 1270 via the I2S bus to enable communication between the processor 1210 and the audio module 1270. In some embodiments, the audio module 1270 can transmit audio signals to the wireless communication module 1260 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0247] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 1270 and the wireless communication module 1260 can be coupled via the PCM bus interface. In some embodiments, the audio module 1270 can also transmit audio signals to the wireless communication module 1260 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0248] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 1210 and the wireless communication module 1260. For example, the processor 1210 communicates with the Bluetooth module in the wireless communication module 1260 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 1270 can transmit audio signals to the wireless communication module 1260 via the UART interface to enable music playback through Bluetooth headphones.

[0249] The MIPI interface can be used to connect the processor 1210 to peripheral devices such as the display screen 1294 and the camera 1293. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 1210 and the camera 1293 communicate via the CSI interface to enable the electronic device's shooting function. The processor 1210 and the display screen 1294 communicate via the DSI interface to enable the electronic device's display function.

[0250] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 1210 to a camera 1293, a display screen 1294, a wireless communication module 1260, an audio module 1270, a sensor module 1280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0251] USB port 1230 is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 1230 can be used to connect a charger to charge electronic devices, and can also be used for data transfer between electronic devices and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0252] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0253] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 1250, wireless communication module 1260, modem processor and baseband processor, etc.

[0254] The wireless communication module 1260 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 1260 can be one or more devices integrating at least one communication processing module. The wireless communication module 1260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 1210. The wireless communication module 1260 can also receive signals to be transmitted from processor 1210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0255] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 1250, and antenna 2 is coupled to wireless communication module 1260, enabling the electronic device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0256] Electronic devices implement display functions through a GPU, a display screen 1294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 1294 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 1210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0257] The display screen 1294 is used to display images, videos, etc. The display screen 1294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 1294, where N is a positive integer greater than 1.

[0258] Electronic devices can achieve shooting functions through ISP, camera 1293, video codec, GPU, display 1294 and application processor.

[0259] The ISP (Image Signal Processor) is used to process data fed back from the camera 1293. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 1293.

[0260] Camera 1293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 1293, where N is a positive integer greater than 1.

[0261] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0262] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0263] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0264] The external storage interface 1220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 1210 through the external storage interface 1220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0265] Internal memory 1221 can be used to store computer executable program code, which includes instructions. Processor 1210 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 1221. Internal memory 1221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, internal memory 1221 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0266] Electronic devices can implement audio functions, such as music playback and recording, through audio modules 1270, speakers 1270A, receivers 1270B, microphones 1270C, headphone jacks 1270D, and application processors.

[0267] The audio module 1270 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 1270 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 1270 may be located in the processor 1210, or some functional modules of the audio module 1270 may be located in the processor 1210.

[0268] The 1270A speaker, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic devices can listen to music or make hands-free calls through the 1270A speaker.

[0269] The receiver 1270B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When an electronic device answers a phone call or voice message, the receiver 1270B can be brought close to the ear to hear the voice.

[0270] Microphone 1270C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 1270C, inputting the sound signal into microphone 1270C. Electronic devices can have at least one microphone 1270C. In some embodiments, electronic devices can have two microphones 1270C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic devices can have three, four, or more microphones 1270C, enabling sound signal collection, noise reduction, sound source identification, and directional recording, among other functions.

[0271] The 1270D headphone jack is used to connect wired headphones. The 1270D headphone jack can be a USB 1230 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0272] Pressure sensor 1280A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 1280A can be disposed on display screen 1294. There are many types of pressure sensors 1280A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 1280A, the capacitance between the electrodes changes. The electronic device determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 1294, the electronic device detects the intensity of the touch operation based on pressure sensor 1280A. The electronic device can also calculate the touch position based on the detection signal from pressure sensor 1280A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0273] The gyroscope sensor 1280B can be used to determine the motion attitude of an electronic device. In some embodiments, the gyroscope sensor 1280B can determine the angular velocity of the electronic device around three axes (i.e., the x, y, and z axes). The gyroscope sensor 1280B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 1280B detects the angle of the electronic device's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 1280B can also be used in navigation and motion-sensing gaming scenarios.

[0274] The barometric pressure sensor 1280C is used to measure air pressure. In some embodiments, the electronic device calculates altitude using the air pressure value measured by the barometric pressure sensor 1280C to assist in positioning and navigation.

[0275] The magnetic sensor 1280D includes a Hall effect sensor. The electronic device can use the magnetic sensor 1280D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device is a flip phone, the electronic device can detect the opening and closing of the flip cover using the magnetic sensor 1280D. Furthermore, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be configured.

[0276] The 1280E accelerometer sensor can detect the magnitude of acceleration in various directions (typically three axes) of electronic devices. When the electronic device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices, and is applicable to screen orientation switching, pedometers, and other applications.

[0277] The 1280F distance sensor is used to measure distance. Electronic devices can measure distance using infrared or laser. In some embodiments, during scene capture, the electronic device can utilize the 1280F distance sensor for distance measurement to achieve fast focusing.

[0278] The proximity sensor 1280G can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED can be an infrared LED. The electronic device emits infrared light outward through the LED. The electronic device uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that an object is near the electronic device. When insufficient reflected light is detected, the electronic device can determine that no object is near the electronic device. The electronic device can use the proximity sensor 1280G to detect when a user holds the electronic device close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 1280G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0279] The ambient light sensor 1280L is used to sense ambient light intensity. Electronic devices can adaptively adjust the brightness of the display 1294 based on the sensed ambient light level. The ambient light sensor 1280L can also be used to automatically adjust white balance when taking photos. The ambient light sensor 1280L can also be used in conjunction with the proximity sensor 1280G to detect whether electronic devices are in a pocket, preventing accidental touches.

[0280] The 1280H fingerprint sensor is used to collect fingerprints. Electronic devices can utilize the collected fingerprint characteristics to achieve fingerprint unlocking, app access locks, fingerprint photography, fingerprint answering of calls, etc.

[0281] Temperature sensor 1280J is used to detect temperature. In some embodiments, the electronic device uses the temperature detected by temperature sensor 1280J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 1280J exceeds a threshold, the electronic device reduces the performance of the processor located near temperature sensor 1280J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, the electronic device heats battery 1242 to prevent abnormal shutdown of the electronic device due to low temperature. In still other embodiments, when the temperature is below yet another threshold, the electronic device boosts the output voltage of battery 1242 to prevent abnormal shutdown due to low temperature.

[0282] Touch sensor 1280K, also known as a "touch panel," can be located on display screen 1294. The touch sensor 1280K and display screen 1294 together form a touchscreen, also known as a "touchscreen." Touch sensor 1280K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 1294. In other embodiments, touch sensor 1280K may also be located on the surface of the electronic device, in a different position than display screen 1294.

[0283] The bone conduction sensor 1280M can acquire vibration signals. In some embodiments, the bone conduction sensor 1280M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 1280M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 1280M can also be incorporated into headphones to form bone conduction headphones. The audio module 1270 can parse the voice signals based on the vibration signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 1280M to realize voice functionality. The application processor can parse heart rate information based on the blood pressure signals acquired by the bone conduction sensor 1280M to realize heart rate detection functionality.

[0284] Buttons 1290 include a power button, volume buttons, etc. Buttons 1290 can be mechanical buttons or touch-sensitive buttons. The electronic device can receive button input and generate key signal inputs related to user settings and function control of the electronic device.

[0285] Motor 1291 can generate vibration alerts. Motor 1291 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to different touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 1291 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 1294. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0286] Indicator 1292 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0287] The SIM card interface 1295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 1295 to achieve contact and separation with the electronic device. The electronic device can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 1295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 1295 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 1295 is also compatible with different types of SIM cards. The SIM card interface 1295 is also compatible with external memory cards. The electronic device interacts with the network through the SIM card to achieve functions such as calls and data communication. In some embodiments, the electronic device uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device and cannot be separated from it.

[0288] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0289] Figure 13 This is a software structure block diagram of an electronic device according to an embodiment of this application.

[0290] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0291] The application layer can include a series of application packages.

[0292] like Figure 13 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0293] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0294] like Figure 13 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0295] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0296] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0297] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0298] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0299] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0300] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0301] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0302] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0303] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0304] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0305] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0306] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0307] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0308] A 2D graphics engine is a graphics engine for 2D drawing.

[0309] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0310] The following example, using a scene of capturing a photograph, illustrates the workflow of the electronic device's software and hardware.

[0311] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a single touch operation as an example, where the corresponding control is the camera application icon, the camera application calls the interface of the application framework layer to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through the camera 1293.

[0312] This application also provides an electronic device, including: a processor for running a computer program stored in a memory to implement one or more steps of any of the methods described above.

[0313] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the above methods.

[0314] This application also provides a computer program product containing instructions. When the computer program product is run on a computer or processor, it causes the computer or processor to perform one or more steps of any of the methods described above.

[0315] This application also provides a chip system including a memory and a processor, the processor executing a computer program stored in the memory to implement one or more steps of any of the methods described above.

[0316] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0317] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0318] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0319] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0320] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0321] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0322] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0323] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0324] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0325] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech parsing method, characterized in that, include: The system acquires voice commands and information issued by running applications, wherein the voice commands are used to instruct the terminal device to perform operations, and the information issued by the application includes reminder information for reminding the user, including voice reminder information. Based on the voice command and the reminder information, the user intent corresponding to the voice command is determined. Specifically, when the application is running in the foreground, the user intent corresponding to the voice command is determined based on the voice command, the reminder information, and the application's interface information. When the application is running in the background, the user intent corresponding to the voice command is determined based on the voice command, the reminder information, and the application's identifier.

2. The method according to claim 1, characterized in that, Before determining the user intent corresponding to the voice command based on the voice command and the reminder information, the method further includes: Obtain a first application list and a second application list, wherein the first application list is a list of all applications installed on the terminal device, and the second application list is a list of applications currently running on the terminal device; Based on the first application list and the second application list, determine the identifier of the application corresponding to the voice command and the running status of the application.

3. The method according to claim 2, characterized in that, The step of obtaining interface information corresponding to the current interface based on the current interface of the application includes: Extract the current interface to obtain the interface content included in the current interface; The interface content is parsed to obtain the interface information corresponding to the application.

4. The method according to any one of claims 1 to 3, characterized in that, The information obtained from acquiring voice commands and the information emitted by the running application includes: The voice command is acquired at the first moment; Based on the first moment, obtain the information emitted by each of the applications that were running within a preset time period before the first moment.

5. The method according to any one of claims 1 to 3, characterized in that, The acquisition of information emitted by the running application includes: Get information sent by the running application in real time.

6. The method according to any one of claims 1 to 3, characterized in that, The acquisition of information emitted by the running application includes: The audio data played by the terminal device is obtained through a pre-set interface; The audio data is converted using Automatic Speech Recognition (ASR) technology to obtain the text information sent by the application.

7. The method according to any one of claims 1 to 3, characterized in that, The acquisition of information emitted by the running application includes: The text data sent by the application is extracted through a pre-set interface to obtain the information in text form sent by the application.

8. The method according to any one of claims 1 to 3, characterized in that, Before determining the user intent corresponding to the voice command based on the voice command and the reminder information, the method further includes: The voice commands are converted using ASR technology to obtain text commands in text form; The step of determining the user intent corresponding to the voice command based on the voice command and the reminder information includes: Based on the text instruction and the reminder information, determine the user intent corresponding to the voice instruction.

9. The method according to claim 8, characterized in that, The process of using ASR technology to convert the voice commands into text commands includes: The voice command is denoised using voice enhancement technology to obtain a denoised voice command. The ASR technology is used to convert the denoised speech command into a text command.

10. The method according to any one of claims 1 to 3, characterized in that, Prior to acquiring voice commands and information emitted by the running application, the method further includes: Based on multiple types of sample data, various correlations are established between different types of sample data. The multiple types of sample data include: sample reminder information, sample interface content, sample voice commands, and sample user intents. The multiple correlations include: the correlation between the sample user intents and the sample reminder information, the correlation between the sample user intents and the sample voice commands, and the correlation between the sample user intents and the sample interface content. Training is performed based on various relationships to obtain a fusion model, which is a single model or a group of models.

11. The method according to claim 10, characterized in that, The step of determining the user intent corresponding to the voice command based on the voice command and the reminder information includes: The user intent corresponding to the voice command is determined by combining the voice command and the reminder information through the fusion model.

12. The method according to any one of claims 1 to 3, characterized in that, After determining the user intent corresponding to the voice command based on the voice command and the reminder information, the method further includes: Based on the user intent, the intent execution interface is invoked to perform an operation that matches the user intent.

13. The method according to any one of claims 1 to 3, characterized in that, The method is applied in a multi-device scenario, which includes a first terminal device and a second terminal device, wherein the first terminal device is connected to the second terminal device. The information obtained from acquiring voice commands and the information emitted by the running application includes: The first terminal device acquires voice commands and information issued by the application running on the first terminal device; The first terminal device sends an information request instruction to the second terminal device according to the voice command. The information request instruction is used to instruct the second terminal device to obtain and provide feedback to the first terminal device on the information sent by the application running on the second terminal device. The first terminal device receives information from the running application fed back by the second terminal device.

14. An electronic device, characterized in that, include: A processor for running a computer program stored in a memory to implement the speech parsing method as described in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the speech parsing method as described in any one of claims 1 to 13.

16. A chip system, characterized in that, The chip system includes a memory and a processor, the processor executing a computer program stored in the memory to implement the speech parsing method as described in any one of claims 1 to 13.

Images