Method for controlling terminal device by means of voice, and terminal device

Abstract

The present application relates to the technical field of terminals. Provided are a method for controlling a terminal device by means of voice and a terminal device. The method comprises: in response to an operation of a user turning on a switch, the terminal device starting a voice control function; and after the voice control function is started, executing a first instruction corresponding to a first voice input by the user, wherein an interface displayed by the terminal device does not comprise a text corresponding to the first instruction, that is, a voice instruction supported by the terminal device is not limited to the text in the interface of the terminal device. The terminal device may support users in controlling the terminal device by means of more, more comprehensive and richer voices.

Description

Method for controlling terminal device by voice and terminal device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 30, 2023, with application number 202310809133.2 and invention name “A method and terminal device for controlling a terminal device by voice”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a method for controlling a terminal device through voice and a terminal device. Background Art

[0003] With the advancement of terminal technology, terminal devices are becoming increasingly common in our daily lives. To improve user efficiency, these devices typically offer a variety of human-computer interaction methods, including a human-computer interface (HMI), voice control, and gesture control. Voice control is a primary and convenient method of HMI. This is particularly useful when users are driving, cooking, reading, or using their hands to control their devices.

[0004] Due to various requirements such as user information security and terminal device power consumption, the voice control function needs to be turned on before it can be used. In some designs, the voice control function is turned on by a wake-up word. The user inputs the wake-up word to the terminal device through voice. After the terminal device is awakened, it can perform corresponding operations according to the user's voice. After the user stops voice input (such as pausing for a few seconds), the voice control function is automatically turned off. Each time the user uses the voice control function, he or she needs to input the wake-up word through voice to wake up the terminal device. If the terminal device needs to be controlled by voice for a long time, the terminal device must be woken up frequently, which is cumbersome.

[0005] Some designs also use a switch to turn the voice control function on or off. Users can turn on the preset switch on the terminal device to enable the voice control function, or turn off the preset switch on the terminal device to disable the voice control function. While the switch is on, the terminal device can execute corresponding commands based on the user's voice input, and the user can conveniently control the terminal device through voice. Currently, the range of voices supported by terminal devices in this scenario is relatively small. For example, it can only support voices that match the text in the display interface; the user experience of using the voice control function is poor.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a method and terminal device for controlling a terminal device through voice. The user does not need to input a wake-up word. As long as the preset switch is turned on, the user can interact with the terminal device through voice at any time. The voice supported by the terminal device is richer and more comprehensive, which improves the user experience of controlling the terminal device through voice.

[0008] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0009] In a first aspect, a method for controlling a terminal device via voice is provided. The method is applied to a terminal device including a microphone. The method comprises: the terminal device displays a first interface, the first interface including a first switch; in response to a user turning on the first switch, the terminal device enables a voice control function. In response to a user turning off the first switch, the terminal device disables the voice control function. The voice control function is enabled or disabled via the first switch and does not require waking up the terminal device. For example, the voice control function is a see-and-speak function. After the voice control function is enabled, upon receiving a first voice input from a user while the terminal device displays a second interface, the terminal device executes a first command corresponding to the first voice input; the second interface does not include a first text corresponding to the first command. In other words, the voice commands supported by the terminal device are not limited to the text within the interface. The terminal device can support users controlling the terminal device via a wider range of voice commands, a more comprehensive range of voice commands, and a richer range of voice commands.

[0010] Enabling the voice control function includes enabling the corresponding recording channel, which is used to capture user voice captured by the microphone. Disabling the voice control function includes disabling the corresponding recording channel. Once the corresponding recording channel is enabled, the voice control function can capture the voice (audio stream) captured by the microphone through the recording channel.

[0011] In conjunction with the first aspect, in one possible implementation, enabling the voice control function on the terminal device includes: the terminal device displaying a first recording icon and a second recording icon. The first recording icon indicates that the voice control function is enabled, allowing the user to conveniently access the voice control function status through the recording icon. The second recording icon indicates that the recording channel is enabled, prompting the user that the terminal device is in the voice recording state, thereby preventing the user's privacy from being disclosed.

[0012] In combination with the first aspect, in a possible implementation, the first voice matches at least one preset hot word.

[0013] In this method, a number of hot words are pre-set in the terminal device, and these hot words are not retrieved from the display interface. If the user's voice input matches any of the hot words, the corresponding command can be executed. Because the hot words are not retrieved from the display interface, the voice commands supported by the terminal device are not limited to the text in the interface. This allows the terminal device to support more comprehensive and richer voice commands.

[0014] In some embodiments, the pre-set hot words include at least one of the following: swipe left, swipe right, swipe up, return to desktop, and return. These hot words are system-level hot words and are applicable to any application on the terminal device.

[0015] In some embodiments, the application to which the second interface belongs belongs to the audio category, the short video category, or the long video category, and the pre-set hot words include at least one of the following: play, pause, stop, fast forward, and fast rewind.

[0016] In some implementations, the application to which the second interface belongs belongs to the e-book playback category, and the pre-set hot words include at least one of the following: previous page, next page, directory, and next chapter.

[0017] In combination with the first aspect, in a possible implementation, before receiving the first voice input by the user when the terminal device displays the second interface, it is detected that the second interface is switched to the foreground display, and the terminal device updates the hot words corresponding to the second interface in the cache; after receiving the first voice input by the user when the terminal device displays the second interface, the terminal device obtains the hot words corresponding to the second interface from the cache.

[0018] In this method, when the interface switches to the foreground, the corresponding hot words in the cache are updated. When the user's voice is received, the hot words corresponding to the interface are directly retrieved from the cache, without the need for real-time analysis from the current display interface. This speeds up the voice recognition rate and the terminal device's response to the user's voice, improving the user experience.

[0019] In conjunction with the first aspect, in one possible implementation, the terminal device updates the hot words corresponding to the second interface in the cache, including: the terminal device obtains the scene-level hot words corresponding to the second interface, and marks the scene-level hot words corresponding to the second interface in the cache. The scene is pre-configured based on the type of application, and the correspondence between the scene and the scene-level hot words is pre-configured.

[0020] In conjunction with the first aspect, in one possible implementation, the cache of the terminal device includes at least one list item, each list item being used to store an interface hot word corresponding to an interface (an interface hot word is applicable to an interface). The terminal device updates the hot word corresponding to the second interface in the cache, including: if the list item corresponding to the second interface does not exist in the cache of the terminal device, the terminal device creates a list item corresponding to the second interface in the cache; the list item corresponding to the second interface is used to store the interface hot word corresponding to the second interface. Then, the terminal device obtains the text included in the second interface; the terminal device saves the interface hot word corresponding to the second interface in the list item corresponding to the second interface based on the text included in the second interface.

[0021] In this method, the cache stores the interface hotwords corresponding to each interface. The cache may include multiple list items, each of which stores the interface hotwords corresponding to a specific interface. This way, when the display interface switches frequently, since the cache stores the interface hotwords corresponding to multiple display interfaces, there is no need to frequently parse text from the display interface again when the display interface switches, reducing the power consumption generated by parsing text and improving the response speed of the terminal device.

[0022] In combination with the first aspect, in a possible implementation, before the terminal device creates a list item corresponding to the second interface in the cache, if the number of list items in the cache is greater than or equal to a preset threshold, one or more list items in the cache are cleared.

[0023] In this method, the number of list items stored in the cache is less than or equal to a preset threshold. This can save storage space and avoid linked list anomalies causing the cache to become too large.

[0024] In combination with the first aspect, in a possible implementation, clearing one or more list items in the cache includes: clearing list items corresponding to one or more interfaces with the shortest total cumulative call time in the cache.

[0025] In conjunction with the first aspect, in one possible implementation, the first speech is the first user speech received after the second interface switches to the foreground. That is, the hotwords corresponding to that interface only need to be retrieved from the cache when the user speech is first received after the interface switches to the foreground. If this is not the first time the user speech is received after the interface switches to the foreground, the speech processing engine can directly parse and recognize the user speech without having to retrieve the hotword set again.

[0026] In this method, since hot words correspond to interfaces, if the interface is not switched, there is no need to re-acquire hot words, which makes the processing flow more concise.

[0027] In combination with the first aspect, in a possible implementation, when the second interface switches to the foreground, the terminal device obtains system-level hot words, scene-level hot words corresponding to the second interface, and interface hot words from the cache after receiving the user's first voice for the first time.

[0028] The terminal device obtains the parsed text of the first voice; and performs speech recognition on the parsed text of the first voice according to the hot words corresponding to the second interface. The terminal device first matches the parsed text of the first voice with the system-level hot words; if the parsed text of the first voice fails to match the system-level hot words, the parsed text of the first voice is matched with the scene-level hot words corresponding to the second interface; if the parsed text of the first voice fails to match the scene-level hot words corresponding to the second interface, the parsed text of the first voice is matched with the interface hot words corresponding to the second interface.

[0029] System-level hotwords take precedence over scene-level hotwords, which in turn take precedence over interface hotwords. This allows speech analysis and recognition to be closer to the user's true intent, preventing misuse.

[0030] In a second aspect, a terminal device is provided, which has the functionality to implement the method described in the first aspect. The functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functionality described above.

[0031] In a third aspect, a terminal device is provided, comprising: a processor, a memory and a microphone; the memory is used to store computer execution instructions, and when the terminal device is running, the processor executes the computer execution instructions stored in the memory to enable the terminal device to execute a method as described in any one of the above-mentioned first aspects.

[0032] In a fourth aspect, a terminal device is provided, comprising: a processor; the processor is used to couple with a memory, and after reading instructions in the memory, execute the method as described in any one of the above-mentioned first aspects according to the instructions.

[0033] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer can execute any one of the methods in the first aspect.

[0034] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any one of the methods described in the first aspect.

[0035] In a seventh aspect, a device (e.g., a chip system) is provided, comprising a processor for supporting a terminal device in implementing the functions described in the first aspect. In one possible design, the device further comprises a memory for storing program instructions and data necessary for the terminal device. When the device is a chip system, it may be composed of a chip or may include a chip and other discrete components.

[0036] Among them, the technical effects brought about by any design method in the second to seventh aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1A is a schematic diagram of a scenario example of a method for controlling a terminal device by voice;

[0038] FIG1B is a schematic diagram of another scenario example of a method for controlling a terminal device by voice;

[0039] FIG2A is a schematic diagram of another scenario example of a method for controlling a terminal device by voice;

[0040] FIG2B is a second schematic diagram of another scenario example of a method for controlling a terminal device by voice;

[0041] FIG3 is a flow chart of a method for controlling a terminal device by voice;

[0042] FIG4A is a schematic diagram of a first scenario example of a method for controlling a terminal device by voice according to an embodiment of the present application;

[0043] FIG4B is a second schematic diagram of a scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0044] FIG5 is a third schematic diagram of a scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0045] FIG6 is a fourth schematic diagram of a scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0046] FIG7 is a fifth schematic diagram of a scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0047] FIG8 is a sixth schematic diagram of a scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0048] FIG9 is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0049] FIG10 is a schematic diagram of a software architecture of a terminal device provided in an embodiment of the present application;

[0050] FIG11 is a first schematic diagram of the interaction between various modules of a terminal device when implementing the method for controlling a terminal device by voice provided in an embodiment of the present application;

[0051] FIG12 is a second schematic diagram of the interaction between the modules of the terminal device when implementing the method for controlling the terminal device by voice provided in an embodiment of the present application;

[0052] FIG13 is a flow chart of a method for controlling a terminal device by voice according to an embodiment of the present application;

[0053] FIG14 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0054] FIG15 is a schematic diagram of the structural composition of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0056] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the situation where A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0057] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0058] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0059] Voice control function:

[0060] Many devices support voice control. These devices use microphones to capture user voice, analyze and recognize it, and execute commands corresponding to the user's voice, enabling users to control the device through voice.

[0061] Generally speaking, in order to save power consumption of terminal devices and avoid false triggering, the voice control function needs to be enabled before it can be used. For example, the user inputs a preset word (called a wake-up word) into the terminal device through voice to wake up the terminal device. After the terminal device is awakened, it can execute the command corresponding to the user's voice, which means that the voice control function is enabled. For example, the user can turn on or off the voice control function by turning on or off a preset switch in the human-computer interaction interface of the terminal device.

[0062] The voice control function may have different names on different terminal devices, such as "Voice Control," "Smart Voice," "Voice Assistant," "See and Speak," "Voice Command," "Free Command," and "Smart AI." The specific implementation of voice control functions with different names may also differ.

[0063] The following examples illustrate several different implementations of the voice control function.

[0064] Voice Assistant:

[0065] Before the user uses the voice assistant to control the terminal device, the terminal device needs to be woken up first. In one example, the wake-up word input by the user to the terminal device is detected, and the terminal device is awakened. In another example, the terminal device is awakened by detecting the user's long press of the power button. In yet another example, the terminal device is awakened by detecting the breath generated when the user inputs voice to the terminal device. Generally speaking, before the terminal device is awakened, the microphone of the terminal device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone is only detected for wake-up words at the kernel layer, and the corresponding recording channel of the voice assistant is not started in the system and driver of the terminal device.

[0066] In response to a user action (e.g., receiving a wake-up word via user voice input), the terminal device wakes up, and the system and driver activate the corresponding recording channel for the voice assistant. After the terminal device wakes up, the voice (audio stream) collected by the microphone is sent to the voice assistant application for processing via the corresponding recording channel of the voice assistant. In this way, the terminal device can execute the instructions corresponding to the user's voice, enabling the user to control the terminal device through voice; it can also realize functions such as dialogue with the user.

[0067] For example, Figure 1A shows a schematic diagram of a scenario example in which a user uses a voice assistant to control a mobile phone. As shown in Figure 1A, the mobile phone displays the desktop interface, and the user inputs the voice "Hello YOYO" into the mobile phone. In response to receiving the wake-up word "Hello YOYO", the mobile phone is awakened. For example, after the mobile phone is awakened, the voice "I'm here" is played to prompt the user that the mobile phone has been awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. For example, as shown in Figure 1A, the user inputs the voice "open video" into the mobile phone, the mobile phone parses and recognizes the user's voice, and executes the instruction corresponding to the voice "open video". For example, in response to receiving the user's voice "open video", the mobile phone starts the video application.

[0068] For example, Figure 1B shows a schematic diagram of another scenario example in which a user uses a voice assistant to control a mobile phone. As shown in Figure 1B, the mobile phone displays the desktop interface, and the user long-presses the power button. In response to receiving the user's long-press operation of the power button, the mobile phone is awakened. After the mobile phone is awakened, the user can control the mobile phone through voice. For example, as shown in Figure 1B, the user inputs the voice "open video" into the mobile phone, and the mobile phone parses and recognizes the user's voice and executes the instruction corresponding to the voice "open video". For example, in response to receiving the user's voice "open video", the mobile phone starts the video application.

[0069] In some implementations, after the terminal device is awakened, the user can give a command to the terminal device by inputting voice into the terminal device, and the terminal device executes the command corresponding to the voice. After the terminal device executes a command, or if the terminal device does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after being awakened, the terminal device will no longer respond to the command given by the user through voice. For example, the terminal device will turn off the recording channel corresponding to the voice assistant. The user needs to input the wake-up word into the terminal device again to wake up the terminal device before he can give commands to the terminal device by inputting voice into the terminal device again. In other words, after the terminal device is awakened, it enters a "short reception" state and can respond to the command given by the user through voice within a short period of time (for example, within 8 seconds).

[0070] In some implementations, when a terminal device is connected to a network, it can wake up and enter a continuous conversation scenario, allowing the user to continue the conversation with the terminal device. After each announcement, the terminal device will continue to pick up the audio without having to wake up again until the user exits the continuous conversation using a command such as "Exit".

[0071] You can see that:

[0072] It can be seen that it is implemented locally by the terminal device and does not require a network connection.

[0073] Visible and Talkable is controlled by a preset switch. Users can turn on Visible and Talkable by turning on the preset switch, and can also turn off the preset switch to stop Visible and Talkable.

[0074] Taking a mobile phone as an example, as shown in FIG2A , a user can open the settings function of mobile phone 100 by, for example, clicking the "Settings" application icon on the desktop. In response to the user clicking the "Settings" application icon, mobile phone 100 displays "Settings" interface 101. "Settings" interface 101 includes a "Smart Voice" option 102, which is used to set up the Smart Voice function. In response to the user clicking "Smart Voice" option 102, mobile phone 100 displays "Smart Voice" interface 103. "Smart Voice" interface 103 includes a "See and Speak" option 104. The user can click "See and Speak" option 104 to set options related to the See and Speak function. In response to the user clicking "See and Speak" option 104, mobile phone 100 displays "See and Speak" interface 105. Optionally, "See and Speak" interface 105 includes a prompt 106 that prompts the user on how to use the See and Speak function. "See and Speak" interface 105 also includes a "See and Speak" switch 107. The user can click "See and Speak" switch 107 to turn the "See and Speak" switch on or off. In one example, in response to receiving a user click on "See and Speak" switch 107, the "See and Speak" switch of mobile phone 100 is turned on, enabling the "See and Speak" feature. Optionally, "See and Speak" interface 105 displays a prompt message 108 to inform the user that the "See and Speak" feature has been successfully enabled.

[0075] In one embodiment, after the "See and Talk" function is turned on, the mobile phone 100 displays a first recording icon, which indicates that the "See and Talk" function is turned on. For example, as shown in FIG2A , after the "See and Talk" switch 107 is turned on, the status bar of the mobile phone 100 display interface displays a recording icon 10a, indicating that the "See and Talk" function is turned on.

[0076] In one scenario, the preset switch for Visible and Talk on the terminal device is not turned on, and the microphone of the terminal device is not turned on. When the preset switch for Visible and Talk is turned on, the terminal device starts the microphone and starts the corresponding recording channel for Visible and Talk in the system and driver. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talk application for processing through the corresponding recording channel of Visible and Talk, allowing users to control the terminal device through voice.

[0077] In another scenario, the preset switch corresponding to the Visible and Talk function of the terminal device is not turned on, and the microphone of the terminal device operates in a power-saving mode (for example, searching for signals at a lower power) to pick up sounds from the surrounding environment. When the preset switch corresponding to the Visible and Talk function is turned on, the system and driver of the terminal device start the corresponding recording channel of the Visible and Talk function. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talk application for processing through the corresponding recording channel of the Visible and Talk function, allowing users to control the terminal device through voice.

[0078] When Visible and Talking is enabled, the corresponding recording channel is activated in the terminal device's system and driver. The terminal device enters a "long reception" state, continuously collecting surrounding sounds. Users can give commands to the terminal device at any time through voice, without having to enter a wake-up word to wake the terminal.

[0079] In one embodiment, after any function on the terminal device turns on the voice recording function of the terminal device (turns on the microphone and turns on the recording channel), the terminal device will send a prompt message to the user to remind the user that the terminal device is in the voice collection state. In this way, the user's privacy can be avoided from being leaked. For example, after the visible and speaking function is turned on, the terminal device enters a continuous voice collection state, and a second recording icon is displayed on the terminal device display interface. The second recording icon indicates that the recording channel is turned on, which is used to remind the user that the microphone is collecting voice. For example, as shown in Figure 2A, the status bar of the mobile phone 100 display interface displays a recording icon 10b, indicating that the recording channel is turned on.

[0080] When the "See and Speak" feature is enabled, the device starts recording and continuously captures the user's voice through the microphone. Users can input voice into the device at any time. The device analyzes and recognizes the user's voice input and executes the corresponding command.

[0081] In one example, as shown in FIG2B , the mobile phone 100 displays a desktop interface. The user inputs a voice message “open video” to the mobile phone 100. In response to receiving the user input voice message “open video”, the mobile phone 100 starts a video application and displays the user interface 120 of the video application.

[0082] Hot words:

[0083] The terminal device can be set with some hot words. After it is turned on, if the received user voice matches at least one of the set hot words, the terminal device will execute the instruction corresponding to the user voice.

[0084] Hot words may be pre-configured on the terminal device, obtained based on the content displayed on the terminal device interface, or input by the user.

[0085] In one implementation, each time an application switch or an in-application interface switch is detected, the terminal device retrieves the text information of the current foreground application interface and sets a hotword based on the retrieved text information. In this way, the user can input a voice message that matches the hotword to control the terminal device to execute the corresponding instruction.

[0086] Exemplarily, referring to Figure 3, the user turns on the switch of "See and Speak" on the terminal device, and turns on "See and Speak". After that, application 1 switches to the foreground to run. The voice control module obtains the elements (controls, etc.) in the display interface of application 1, obtains the interface information (text included in the display interface), and refreshes the saved interface information. After that, application 2 switches to the foreground to run. The voice control module obtains the elements (controls, etc.) in the display interface of application 2 and refreshes the saved interface information. The user inputs voice to the terminal device, and the voice control module sets hot words according to the saved interface information. If it is determined that the received user voice matches at least one hot word, the instruction corresponding to the user voice is executed.

[0087] For example, referring to the scenario shown in FIG2B , the mobile phone displays the desktop interface. After detecting that the desktop application has switched to the foreground, the mobile phone obtains and saves the text information in the desktop interface, including "5G", "8:00", "Clock", "Calendar", "Gallery", "Memo", "File Management", "Email", "Music", "Calculator", "App Store", "Sports Health", "Weather", "Browser", "Settings", "AR Application", "Video", "Camera", "Contact Book", "Phone", "Message", etc. The text information "5G", "8:00", "Clock", "Calendar", "Gallery", "Memo", "File Management", "Email", "Music", "Calculator", "App Store", "Sports Health", "Weather", "Browser", "Settings", "AR Application", "Video", "Camera", "Contact Book", "Phone", "Message", etc. are set as hot words.

[0088] The user enters the voice command "Open Video" into the phone. After receiving the user's voice command, the phone parses and recognizes the user's voice command, determines that the voice command "Open Video" contains the hot word "video," and therefore determines that the voice command "Open Video" matches the hot word "video." The phone then executes the command corresponding to the voice command "Open Video." For example, as shown in FIG2B , in response to receiving the user's voice command "Open Video," the phone launches the video application and displays the video application's user interface.

[0089] The video app switches to the foreground, and the phone retrieves and saves the text information displayed on the video app's display interface, including "5G," "8:00," "media title," "selection," "1," "2," "3," "4," "5," "6," "7," "8," and "9." "5G," "8:00," "media title," "selection," "1," "2," "3," "4," "5," "6," "7," "8," and "9" are set as hot words. The user inputs voice into the phone, and if the user's voice matches any of the hot words, the phone executes the corresponding command.

[0090] In this implementation, each time an application is switched or the display interface within the application is switched, the terminal device will update the hot words according to the text information of the current display interface. The hot words are set and updated according to the text in the current display interface of the terminal device. Text that is not included in the display interface will not be set as a hot word. The scenarios for obtaining hot words are relatively limited. In this way, the voices supported by the terminal device are relatively limited and the types are relatively small. For example, when the mobile phone displays the video application display interface in Figure 2B, if the user inputs the voice "pause", the mobile phone confirms that the voice "pause" does not match all the hot words and will not execute the corresponding instruction.

[0091] The present application provides a method for controlling a terminal device via voice. This method supports setting not only text in the display interface as a hotword, but also text not included in the display interface as a hotword. The acquisition of hotwords is independent of the text in the display interface. Thus, after enabling "see and speak," the terminal device can support users controlling the terminal device through a wider range of voice functions, more comprehensive functions, and richer voice functions.

[0092] When the "See and Speak" feature is enabled, the device enters a continuous audio reception state. Users can input voice messages into the device. The device parses and recognizes the user's voice input. If it matches any of the hotwords, it executes the corresponding command.

[0093] For example, referring to FIG4A , mobile phone 100 displays the short video playback interface 121 of a short video application and receives a user voice input of "return to desktop." The text information in short video playback interface 121 includes "5G," "8:00," "Follow," "Recommend," "Nearby," "Home," "Friends," "Message," "Me," etc., but does not include "return to desktop." In the method for controlling a terminal device via voice provided in the embodiments of the present application, the acquisition of hot words is independent of the text in the display interface. "Return to desktop" can be set as a hot word. When mobile phone 100 recognizes that the user's voice input "return to desktop" matches the hot word "return to desktop," it executes the command corresponding to the voice input of "return to desktop." In one example, in response to receiving the user's voice input of "return to desktop," the desktop application of mobile phone 100 switches to the foreground, and mobile phone 100 displays the desktop interface. In this example, the hot word "return to desktop" is not obtained from the mobile phone display interface. The acquisition of the hot word is independent of the text in the display interface; the hot word can be pre-configured on the mobile phone.

[0094] In the method for controlling a terminal device via voice provided in an embodiment of the present application, some hot words are pre-set in the terminal device. These hot words are not obtained from the display interface. In one implementation, the terminal device also obtains some hot words from the display interface. In other words, on the terminal device, some hot words come from the text in the display interface, while other hot words are pre-set. In this way, the terminal device supports more and more comprehensive hot words; accordingly, the terminal device can support users to control the terminal device through more, more comprehensive, and richer voice.

[0095] In some embodiments, system-level hotwords are pre-set in the terminal device and apply to all applications on the terminal device. When any application on the terminal device is running in the foreground, if the user's voice input matches at least one system-level hotword, the terminal device executes the instruction corresponding to the user's voice input. System-level hotwords are global and independent of the application or application interface. In one example, system-level hotwords may include: swipe left, swipe right, swipe up, return to desktop, return, etc.

[0096] For example, referring to FIG4A , when the mobile phone 100 displays the interface of the short video application, the mobile phone 100 may respond to the user's voice input of “return to desktop” and execute the return to desktop instruction.

[0097] In another example, as shown in FIG4B , the mobile phone 100 displays the application interface 122 of the e-book application and receives the user's voice input "return to desktop." The mobile phone 100 recognizes that the user's voice input "return to desktop" matches the hot word "return to desktop" and executes the command corresponding to the voice input "return to desktop." For example, as shown in FIG4B , the desktop application of the mobile phone 100 switches to the foreground and the mobile phone 100 displays the desktop interface.

[0098] It should be noted that the mobile phone 100 can perform semantic understanding of the user's voice. In other words, the user's voice may not be completely consistent with the hot word, as long as the semantics are the same or close. That is, the user's voice and the hot word are matched, but they do not have to be consistent. For example, in the scenario shown in Figure 4B, the user input voice is "return to desktop", and the system-level hot words include "return to desktop". "Return to desktop" and "return to desktop" have the same semantics, that is, the two match.

[0099] In another example, as shown in FIG5 , the mobile phone 100 displays the desktop interface and receives the user's voice input "swipe left." The mobile phone 100 recognizes that the user's voice input "swipe left" matches the system-level hotword "swipe left" and executes the command corresponding to the voice input "swipe left." For example, as shown in FIG5 , the mobile phone 100 displays the negative one screen interface.

[0100] In some embodiments, scenario-level hot words are pre-set in the terminal device, and the scenario-level hot words apply to all applications within a scenario. In one implementation, multiple scenarios are pre-set in the terminal device, and each scenario corresponds to at least one application. Exemplary pre-set scenarios may include audio and video scenarios, e-book scenarios, home screen scenarios, permission pop-up scenarios, etc.

[0101] In one example, the correspondence between applications and scenarios is preconfigured in the terminal device. For example, the correspondence between applications and scenarios is shown in Table 1. The terminal device can determine the scenario corresponding to the application based on the application name (application package name).

[0102] Table 1

[0103] In another example, the terminal device pre-configures the correspondence between application types and scenarios. For example, the correspondence between application types and scenarios is shown in Table 2. The terminal device may also pre-configure the correspondence between applications and application types. For example, the correspondence between applications and application types is shown in Table 3. The terminal device may determine the application type based on the application name and determine the scenario corresponding to the application based on the correspondence between the application type and scenario.

[0104] Table 2

[0105] Table 3

[0106] It should be noted that Tables 1 to 3 above are only illustrative of the correspondence between applications and scenarios. In other embodiments, the terminal device may obtain the correspondence between applications and scenarios in other ways; for example, the correspondence between applications and scenarios may not be in the form of a configuration table; for example, the correspondence between applications and scenarios may be different from that in the above examples. In other embodiments, the terminal device may not save the correspondence between applications and scenarios; for example, the terminal device may obtain the correspondence between applications and scenarios from a server. The embodiments of the present application do not limit the specific manner in which the terminal device obtains the scenario to which the application belongs.

[0107] Scenario-level hot words are related to the scenario to which the application belongs and are not dependent on the text contained in the application interface. In one example, the scenario-level hot words corresponding to each scenario are shown in Table 4.

[0108] Table 4

[0109] It is understandable that in other embodiments, the scene division method may be different from the above example; for example, more or fewer scenes may be divided. The scene-level hot words corresponding to each scene may also be adjusted according to actual conditions.

[0110] When any application on the terminal device is running in the foreground, in response to receiving the user's voice input, the terminal device determines the scene to which the application belongs. If the user's voice input matches at least one scene-level hot word corresponding to the scene to which the application belongs, the terminal device executes the instruction corresponding to the user's voice.

[0111] For example, FIG6 shows a schematic diagram of an example scenario in which a mobile phone receives user voice in an audio and video scenario.

[0112] 6, the mobile phone 100 displays a short video application (eg, ) of the short video playback interface 121. The scenario to which the short video application belongs is the audio and video scenario, and the scene-level hot words corresponding to the audio and video scenario include: play, pause, stop, fast forward, fast rewind, etc. The user inputs the voice "pause" to the mobile phone 100. The voice "pause" input by the user is received, and the voice "pause" matches the hot word "pause". In response to receiving the voice "pause", the mobile phone 100 executes the instruction corresponding to the voice "pause" to pause the playback of the short video. In one example, as shown in Figure 6, a "play" button 123 is displayed on the short video playback interface 121 to prompt the user that the short video has been paused. The user can click the "play" button 123 to continue playing the short video. The user can also input the voice "play" to the mobile phone 100 to continue playing the short video.

[0113] For example, FIG7 shows a schematic diagram of an example scenario in which a mobile phone receives user voice in an e-book scenario.

[0114] 7, the mobile phone 100 displays an e-book application (eg, QQ ) application interface 122. The scene to which the e-book application belongs is the e-book scene, and the scene-level hot words corresponding to the e-book scene include: previous page, next page, directory, next chapter, etc. The user inputs the voice "next page" to the mobile phone 100. The voice "next page" input by the user is received, and the voice "next page" matches the hot word "next page". In response to receiving the voice "next page", the mobile phone 100 executes the instruction corresponding to the voice "next page" and displays the e-book application (for example, QQ ) application interface 124. Wherein, the page number of the application interface 124 is page 20, the page number of the application interface 122 is page 19, and the application interface 124 is the next page of the application interface 122.

[0115] In some embodiments, the terminal device also obtains hot words from the display interface of the foreground application, that is, obtains interface hot words. For example, referring to FIG8 , the mobile phone 100 displays a short video application (for example, ) short video playback interface 121. The text information in the short video playback interface 121 includes "5G", "8:00", "Follow", "Recommend", "Nearby", "Home", "Friends", "Message", "Me", etc. The mobile phone 100 sets the text in the current display interface of the foreground application as interface hot words, that is, the interface hot words include "5G", "8:00", "Follow", "Recommend", "Nearby", "Home", "Friends", "Message", "Me", etc.

[0116] The mobile phone 100 receives the user's voice input "check message", which matches the interface hot word "message", and executes the command corresponding to the voice "check message". For example, as shown in FIG8 , the mobile phone 100 displays the message interface 125.

[0117] The method for controlling a terminal device through voice provided in an embodiment of the present application supports pre-set system-level hot words, scene-level hot words, and interface hot words obtained from the foreground application display interface. The supported hot words are more diverse and comprehensive, and accordingly, users can be supported to control the terminal device through more, more comprehensive, and richer voices.

[0118] The method for controlling a terminal device by voice provided in the embodiments of the present application can be applied to terminal devices that support voice input. The above-mentioned terminal devices may include mobile phones, tablet computers, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (such as smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (such as smart watches, smart bracelets, etc.), vehicle-mounted devices, virtual reality devices, etc., and the embodiments of the present application do not impose any restrictions on this.

[0119] For example, FIG9 is a schematic diagram of the structure of a terminal device 100. The terminal device 100 may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display 294, and a subscriber identification module (SIM) card interface 295. The sensor module 280 may include a temperature sensor, an ambient light sensor, a touch sensor, and the like.

[0120] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device 100. In other embodiments, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0121] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0122] The controller can be the nerve center and command center of the terminal device 100. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0123] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0124] In some embodiments, the processor 210 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0125] It is understood that the interface connection relationship between the modules illustrated in this embodiment is only for illustrative purposes and does not constitute a structural limitation on the terminal device. In other embodiments, the terminal device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0126] The charging management module 240 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 can receive charging input from the wired charger via the USB interface 230. In some wireless charging embodiments, the charging management module 240 can receive wireless charging input via the terminal device's wireless charging coil. While charging the battery 242, the charging management module 240 can also provide power to the terminal device via the power management module 241.

[0127] The power management module 241 is used to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 and provides power to the processor 210, the internal memory 221, the external memory, the display 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 241 can also be set in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

[0128] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0129] The terminal device 100 implements display functions through a GPU, display screen 294, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0130] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0131] The terminal device 100 can realize the shooting function through the ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0132] The ISP processes data fed back by camera 293. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 293.

[0133] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0134] Digital signal processors are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when a terminal device selects a frequency point, the digital signal processor performs a Fourier transform on the frequency point energy.

[0135] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. This allows the terminal device to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0136] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in terminal devices, such as image recognition, face recognition, speech recognition, and text comprehension.

[0137] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0138] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be set in the processor 210, or some functional modules of the audio module 270 can be set in the processor 210. The speaker 270A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 270B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. The microphone 270C, also known as the "microphone" or "microphone", is used to convert sound signals into electrical signals. The headphone jack 270D is used to connect wired headphones. The headphone jack 270D can be a USB port 230, or it can be a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface. In the embodiment of the present application, after the voice control function is turned on, the terminal device 100 continuously collects surrounding sounds through the microphone 270C.

[0139] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 210 via the external memory interface 220 to implement data storage functions. For example, audio, video, and other files can be stored on the external memory card.

[0140] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the terminal device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 can execute instructions stored in the internal memory 221, and the internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data (such as video files) created during the use of the terminal device, etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0141] The buttons 290 include a power button, a volume button, etc. The button 290 can be a mechanical button. It can also be a touch button. The motor 291 can generate a vibration prompt. The motor 291 can be used for an incoming call vibration prompt, or for touch vibration feedback. The indicator 292 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 295, or pulled out from the SIM card interface 295 to achieve contact and separation with the terminal device. The terminal device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0142] In the embodiment of the present application, the terminal device 100 is a terminal device that can run an operating system and install applications. Optionally, the operating system running on the terminal device 100 can be system, system, System, etc.

[0143] In some embodiments, the software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture or a cloud architecture. Taking the system as an example, the software structure of the terminal device 100 is exemplarily described.

[0144] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0145] The application layer may include a series of application packages APK. For example, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications. In the embodiment of the present application, the application layer includes a voice control APK, which is used to provide the voice control function of the terminal device 100. The voice control APK includes an interface monitoring module, a cache management module, an application interaction module, etc. Among them, the interface monitoring module is used to monitor the startup, exit, switching and other processes of the application interface. The cache management module is used to manage the acquisition and caching of hot words. The application interaction module is used to manage the process of the application executing instructions according to voice.

[0146] The application layer also includes a speech processing engine for parsing, recognizing, and processing speech. The speech parsing module converts speech into text; the speech recognition module understands and identifies the semantics of the text and determines whether it matches hot words; and the instruction mapping module converts the recognized semantics into machine-executable instructions.

[0147] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0148] As shown in Figure 10, the application framework layer may include a window manager, a content provider, a view system, a resource manager, an activity manager service (AMS), an application manager (PMS), an interface acquisition and parsing module, a multi-mode control module, etc.

[0149] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0150] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0151] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0152] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0153] The AMS is primarily responsible for starting, switching, and scheduling the four major components of the system, as well as managing and scheduling application processes. Its responsibilities are similar to those of the process management and scheduling module in an operating system. When a process or component is started, the request is passed to the AMS via the Binder communication mechanism, and the AMS then handles it uniformly.

[0154] PMS handles package management related tasks, such as application installation and uninstallation. It is also used to provide all the information of an application, such as startup and shutdown.

[0155] The interface acquisition and parsing module is used to parse elements on the display interface of the terminal device 100, such as controls, text on controls, etc.

[0156] The multi-mode control module is used to manage the execution of voice commands, for example, sending commands to applications so that the applications can execute the commands.

[0157] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0158] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0159] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0160] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0161] A 2D graphics engine is a drawing engine for 2D drawings.

[0162] The Android runtime is responsible for scheduling and management of the Android system. The Android runtime includes the core library and the virtual machine.

[0163] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0164] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0165] The kernel layer is the layer between hardware and software. The kernel layer may include display drivers, sensor drivers, microphone drivers, Wi-Fi drivers, etc.

[0166] In combination with Figure 10, Figure 11 shows a schematic diagram of the interaction of various modules of the terminal device when implementing the method for controlling the terminal device by voice provided in an embodiment of the present application.

[0167] The method for controlling a terminal device via voice provided in this embodiment pre-sets system-level hotwords and scenario-level hotwords for multiple scenarios (such as audio and video scenarios, e-book scenarios, home screen scenarios, and permission pop-up scenarios). For example, system-level hotwords include: swipe left, swipe right, swipe up, return to desktop, and return; the scenario-level hotwords corresponding to each scenario are shown in Figure 4.

[0168] System-level hot words are global and do not depend on applications or application interfaces. In one implementation, system-level hot words are stored in the terminal device and can be applied to various applications.

[0169] Scenario-level hot words apply to all applications within a scenario. In other words, by determining the scenario to which an application belongs, you can obtain the scenario-level hot words corresponding to that application.

[0170] The terminal device can also obtain the text in the interface according to the display interface of the foreground application, that is, obtain the interface hot words.

[0171] In other words, both scene-level hot words and interface hot words are related to applications. When an application is started or switched to the foreground, the terminal device obtains the scene-level hot words and interface hot words corresponding to the application.

[0172] 11 , in some embodiments, in response to a user operation or a call between applications, a terminal device switches between applications or switches interfaces within an application. The interface monitoring module of the voice control APK monitors the application switching or the display interface switching within the application.

[0173] In one implementation, as shown in FIG12 , the AMS may monitor the life cycle of the Activity corresponding to each interface of each application on the terminal device, including onCreate, onStart, onResume, onPause, onStop, onRestart, and onDestroy.

[0174] onCreate: Executed when the Activity is first loaded. The onCreate event is executed when the interface is first launched. After the Activity is destroyed, the onCreate event is re-executed when it is reloaded.

[0175] onStart: Executed after the onCreate event. After exiting the interface and re-entering the interface after a period of time (the Activity is not destroyed), the onCreate event will be skipped and the onStart event will be executed directly.

[0176] onResume: Executed after the onStart event; or after the interface is swapped to the background, when the user views the interface again (the Activity has not been destroyed and the onStop event has not been executed), the onCreate and onStart events will be skipped and the onResume event will be executed directly.

[0177] onPause: Called when the interface is switched to the background.

[0178] onStop: Executed after the onPause event. If the user does not return to the screen within a certain period of time, the onStop event of the activity in that screen will be executed. Or if the user directly removes the screen from the current page, the onStop event of the activity in that screen will also be executed.

[0179] onRestart: After the onStop event is executed, if the interface and the application process are not destroyed by the system, the onRestart event of the activity of the interface will be executed when the user re-enters the interface. After the onRestart event, the onCreate event will be skipped and the onStart event will be executed directly.

[0180] onDestroy: Executed when the Activity is destroyed. After executing the onStop event, if you do not return to the interface again, the Activity will be destroyed.

[0181] The interface monitoring module registers with AMS as a listener. When AMS detects changes in the Activity lifecycle, it calls back to notify the interface monitoring module. This notification includes the component name corresponding to the Activity, which can include both the application package name and the Activity name. Based on these changes in the Activity lifecycle, the interface monitoring module can determine whether an application switch or an in-app display interface switch has occurred.

[0182] In one example, Display Interface 1 of Application 1 corresponds to Activity 1, and Display Interface 2 corresponds to Activity 2. AMS monitors Activity 1's onPause event and Activity 2's onResume event, notifying the interface monitoring module of these events. The interface monitoring module determines that Display Interface 1 has been switched to the background based on Activity 1's onPause event and that Display Interface 2 has been switched to the foreground based on Activity 2's onResume event, thus detecting a display interface switch within the application.

[0183] In another example, the display interface 1 of application 1 corresponds to Activity 1, and the display interface 1 of application 2 corresponds to Activity 2. The AMS monitors the onPause event of Activity 1 and the onResume event of Activity 2, and notifies the interface monitoring module of the onPause event of Activity 1 of application 1 and the onResume event of Activity 2 of application 2. The interface monitoring module determines that the display interface 1 of application 1 has been switched to the background based on the onPause event of Activity 1 of application 1, and determines that the display interface 1 of application 2 has been switched to the foreground based on the onResume event of Activity 2 of application 2; that is, the application switching is detected.

[0184] Continuing with Figure 11, when the interface monitoring module detects an application switch or an in-application display interface switch, it notifies the cache management module to refresh the scene-level hotword cache and the interface hotword cache. In one implementation, the notification message includes the component name of the display interface that has been switched to the foreground. The cache management module can obtain the application package name and Activity name based on the component name.

[0185] In one example, the cache 1 of the terminal device is used to store preset system-level hot words, and the system-level hot words do not need to be updated.

[0186] Cache 2 is used to save scene-level hot words for various preset scenes. The cache management module determines the scene corresponding to the application based on the application package name, and marks the scene-level hot words corresponding to the scene in cache 2. In one implementation, cache 2 saves the scene-level hot words for each scene in the form of a list. For example, List 1 is the scene-level hot words for the audio and video scene, List 2 is the scene-level hot words for the e-book scene, List 3 is the scene-level hot words for the main screen scene, and List 4 is the scene-level hot words for the permission pop-up scene. The status of each list is inactive by default. The cache management module determines the corresponding scene based on the application package name, and sets the list corresponding to the scene in cache 2 to active state, and sets the remaining lists to inactive state. For example, the cache management module determines that the application belongs to the audio and video scene based on the application package name, then sets List 1 to active state, and sets List 2, List 3 and List 4 to inactive state. The list set to active state is the marked list; that is, the scene-level hot words saved in the list are marked. In this way, when the hot word set corresponding to the interface is subsequently determined, the scene-level hot words corresponding to the interface can be easily found from cache 2. There is no need to determine the corresponding scene and scene-level hot words in real time based on the application to which the interface belongs each time the hot word set corresponding to the interface is determined, which improves the processing efficiency of the terminal device in obtaining the hot word set after receiving the user's voice.

[0187] Cache 3 is used to store interface hot words. The cache management module updates cache 3 according to the component name (application package name and Activity name) of the display interface switched to the foreground.

[0188] In one implementation, the cache 3 may store at least one list item, wherein each list item is used to store a group of interface hot words corresponding to an interface.

[0189] For example, a list item includes the following parameters:

[0190] lastUseTime (last usage time), indicating the time when the interface was last called;

[0191] totalUseTime (accumulated usage time), which indicates the total duration of the interface being called;

[0192] cacheId (component identifier), used to uniquely identify a component. For example, cacheId is the component name, including the application package name and Activity name;

[0193] hotword (hot word information), used to save the interface hot words corresponding to this interface.

[0194] In one example, the cache management module matches the component name (application package name and Activity name) corresponding to the display interface switched to the foreground with the cacheId (component identifier) ​​of the list item in cache 3, and searches for the list item corresponding to the display interface switched to the foreground in cache 3.

[0195] If the list item corresponding to the component name does not exist in cache 3, create a list item corresponding to the component name, and update the parameters in the list item accordingly. For example, lastUseTime = current time, totalUseTime = 0, cacheId = component name corresponding to the display interface switched to the foreground. The cache management module calls the interface acquisition and parsing module to obtain the text in the display interface switched to the foreground. In one implementation, all texts in the acquired display interface are set as interface hot words. In another implementation, a part of the text parsed from the display interface is set as an interface hot word. For example, in an e-book scenario, the text related to the control included in the text of the display interface can be set as an interface hot word. Further, set hotword in the newly created list item = interface hot word corresponding to the display interface switched to the foreground.

[0196] If a list item corresponding to the component name exists in cache 3, the lastUseTime parameter in the list item is updated. Exemplarily, lastUseTime = current time. Furthermore, the component name of the previous display interface is obtained, and the corresponding list item is searched in cache 3 based on the component name of the previous display interface. Exemplarily, the cache management module updates the totalUseTime parameter in the list item corresponding to the previous display interface, where totalUseTime = current time - lastUseTime. The previous display interface refers to the display interface displayed before the terminal device displays the current display interface.

[0197] Multiple list items can be saved in cache 3. When a list item is created, the cacheId and hotword of the list item are recorded; when the display interface switches to the foreground, the lastUseTime in the list item corresponding to the display interface is updated; when the display interface exits the foreground, the totalUseTime in the list item corresponding to the display interface is updated. That is to say, in one implementation, the cache management module only needs to call the interface acquisition and parsing module to obtain the text in the display interface switched to the foreground (i.e., obtain the interface hotword corresponding to the display interface switched to the foreground) when creating a list item of a display interface. It is not necessary to call the interface acquisition and parsing module to parse the display interface and obtain the interface hotword corresponding to the display interface every time the display interface switches to the foreground. The power consumption generated by parsing the display interface when the display interface is frequently switched is reduced.

[0198] In one implementation, the number of list items stored in the cache 3 is less than or equal to a preset threshold, which can save storage space and prevent the cache 3 from being too large due to linked list anomalies.

[0199] When a new list item needs to be added to cache 3, the number of list items stored in cache 3 is first determined. If the number of list items stored in cache 3 is greater than or equal to a preset threshold, one or more list items stored in cache 3 are cleared so that the number of list items stored in cache 3 is less than the preset threshold; then the list item is created.

[0200] In one example, one or more list items to be cleared are determined based on the totalUseTime in the list items. For example, N list items with the smallest totalUseTime values ​​(i.e., the shortest total cumulative call duration of the display interface) are cleared; where N is greater than or equal to 1.

[0201] In this method, after monitoring an application switch or an in-application display interface switch, the corresponding scene-level hotword is obtained based on the scene to which the foreground application belongs and marked in cache 2. The corresponding interface hotword is obtained based on the display interface of the foreground application and the corresponding list item in cache 3 is updated. In this way, cache 1, cache 2, and cache 3 can store the system-level hotwords, scene-level hotwords, and interface hotwords corresponding to the display interface switched to the foreground.

[0202] Furthermore, when the terminal device displays the display interface that has been switched to the foreground, the user can input voice into the terminal device. The microphone of the terminal device receives the user's voice and transmits the user's voice to the application interaction module of the voice control APK. The application interaction module transmits the user's voice to the voice processing engine for processing.

[0203] In one implementation, if the user voice is received for the first time after switching to the current display interface, the voice processing engine calls the hot word acquisition module to update the hot word set. In one example, the hot word acquisition module reads the system-level hot words from cache 1, reads the scene-level hot words corresponding to the scene to which the foreground application belongs from cache 2, and reads the interface hot words corresponding to the current display interface of the foreground application from cache 3. The acquired system-level hot words, scene-level hot words, and interface hot words are merged into a hot word set corresponding to the current display interface of the foreground application, and the hot word set is saved in cache 4. Optionally, in another example, since the system-level hot words are applicable to any application, after the hot word set in cache 4 is generated for the first time, there is no need to update the system-level hot words in the hot word set subsequently. If it is an interface switch within the application, or the application belongs to the same scene before and after the switch, there is no need to update the scene-level hot words in the hot word set.

[0204] In one implementation, scene-level hot words are retrieved from the list of active states in cache 2, and interface hot words are retrieved from cache 3 based on the component name. Because scene-level hot words are already cached in cache 2 and interface hot words are cached in cache 3 when the display interface switches to the foreground, when receiving user voice, only the system-level hot words, scene-level hot words, and interface hot words need to be read from the cache, eliminating the need for real-time scene determination and display interface parsing, which improves the response speed during voice input.

[0205] After updating the hotword set, the speech parsing module in the speech processing engine parses the user's speech and obtains the parsed text corresponding to the user's speech. The speech parsing module sends the parsed text to the speech recognition module. The speech recognition module performs semantic understanding and recognition on the parsed text to obtain the recognized content, that is, converting the parsed text into more easily understandable natural language. Exemplarily, the speech recognition module uses a natural language processing (NLP) algorithm to perform semantic analysis on the parsed text.

[0206] In one implementation, when the speech recognition module performs semantic understanding and recognition, it compares the parsed text with the hot word set corresponding to the current display interface of the foreground application stored in cache 4. If the parsed text matches any hot word in the hot word set stored in cache 4, the speech recognition module successfully recognizes it and sends the recognition content to the instruction mapping module.

[0207] When performing semantic understanding and recognition, the parsed text is matched with the hot words. The two can have the same semantics or close semantics or the recognized content contains hot words, etc. It is not necessary for the parsed text to be exactly the same as the hot words. For example, in the scenario shown in Figure 4B, the user input voice is "return to desktop", and the system-level hot words include "return to desktop". "Return to desktop" and "return to desktop" have the same semantics, that is, the two are matched. Since hot words are used in semantic understanding and recognition, semantic recognition has a reference standard, which improves the accuracy of speech recognition and semantic understanding.

[0208] In one implementation, the parsed text is first matched with the system-level hot words in cache 4. If the match is successful (i.e., the parsed text matches any one of the system-level hot words in cache 4), the recognition content is returned. If the parsed text fails to match the system-level hot words in cache 4, the parsed text is matched with the scene-level hot words in cache 4. If the parsed text matches any one of the scene-level hot words in cache 4, the recognition content is returned. If the parsed text fails to match the scene-level hot words in cache 4, the parsed text is matched with the interface hot words in cache 4. If the parsed text matches any one of the interface hot words in cache 4, the recognition content is returned. If the parsed text fails to match the interface hot words in cache 4, the speech recognition fails.

[0209] That is, during speech recognition, the parsed text is prioritized for matching with system-level hotwords. If the match fails with system-level hotwords, it is then matched with scene-level hotwords. If the match fails with scene-level hotwords, it is then matched with interface hotwords. System-level hotwords have a higher priority than scene-level hotwords, which in turn have a higher priority than interface hotwords. This allows speech parsing and recognition to be closer to the user's true intent, avoiding misoperations. For example, in an e-book application, the text in the e-book application's display interface contains the phrase "come to the desktop." When the terminal device receives the user's voice input of "return to the desktop," it first matches the parsed text "return to the desktop" with the system-level hotwords, which include "return to the desktop." This allows the terminal device to execute the command corresponding to "return to the desktop" and display the desktop interface, rather than matching the parsed text "return to the desktop" with the text in the e-book application's display interface.

[0210] Continuing with Figure 11 , the speech recognition module sends the recognition content to the instruction mapping module. The instruction mapping module receives the recognition content and converts it into an instruction that the terminal device can recognize and execute. The instruction mapping module then sends the instruction to the multi-mode control module, which processes the instruction.

[0211] In one implementation, the instruction mapping module simulates and generates corresponding control events based on the recognized content; and sends the control events as instructions to the multi-mode control module. The multi-mode control module sends the control events to the corresponding application for processing. For example, referring to Figure 7, the mobile phone 100 displays the application interface 122 of the e-book application, and the current foreground application is the e-book application. The control "next page" is not set on the application interface 122. After receiving the user voice "next page", the instruction mapping module generates a click event of the control "next page" based on the recognized content "next page". The instruction mapping module sends the click event of the control "next page" to the multi-mode control module, and the multi-mode control module sends the click event of the control "next page" to the e-book application for processing.

[0212] In one implementation, the instruction mapping module sends the instruction to the multi-mode control module, and the multi-mode control module calls the software development kit (SDK) of the relevant application to implement the logic of executing the instruction. For example, still referring to Figure 7, the mobile phone 100 displays the application interface 122 of the e-book application, and the current foreground application is the e-book application. After receiving the user's voice "next page", the instruction mapping module generates a machine instruction to display the next page based on the recognition content "next page". The instruction mapping module sends the machine instruction to display the next page to the multi-mode control module. The multi-mode control module calls the SDK of the e-book application to implement the logic of switching the display interface of the e-book application to the next page of the current display interface.

[0213] In a method for controlling a terminal device via voice provided in an embodiment of the present application, when the terminal device receives user voice, it retrieves a set of hotwords corresponding to the foreground application from a cache. The hotword set includes system-level hotwords, scene-level hotwords corresponding to the foreground application, and interface hotwords corresponding to the display interface of the foreground application. Hotwords include not only text in the display interface, but also system-level hotwords and scene-level hotwords. The terminal device supports more and richer hotwords. Moreover, since the hotwords are retrieved from the cache, there is no need for real-time parsing from the current display interface. This results in a faster voice recognition rate and a faster response speed for the terminal device to the user's voice, improving the user experience. In particular, when the display interface switches back and forth frequently, since the cache stores interface hotwords corresponding to multiple display interfaces, there is no need to frequently parse the text from the display interface again when the display interface switches, reducing the power consumption generated by parsing the text and improving the response speed of the terminal device. If the display interface switches but the scene corresponding to the application remains unchanged, there is no need to update the scene-level hotwords, thereby improving processing efficiency.

[0214] For example, FIG13 shows a flow chart of a method for controlling a terminal device by voice according to an embodiment of the present application. As shown in FIG13 , the method includes:

[0215] The interface monitoring module of the terminal device monitors the application switching or the display interface switching within the application, and the interface monitoring module notifies the cache management module to update the cache. The cache management module obtains the list item corresponding to the previous display interface in cache 3 based on the component name of the previous display interface of the current display interface, and updates the cumulative usage time (totalUseTime) in the list item corresponding to the previous display interface in the cache (cache 3). The cache management module identifies the scene corresponding to the foreground application, and if the scene changes, marks the scene-level hot words corresponding to the foreground application in cache 2. If the scene has not changed, cache 2 does not need to be updated. The cache management module determines whether there is a list item corresponding to the current display interface in the cache (cache 3). If there is a list item corresponding to the current display interface in the cache (cache 3), update the most recent usage time (lastUseTime) in the list item corresponding to the current display interface in the cache (cache 3); if there is no list item corresponding to the current display interface in the cache (cache 3), first determine whether the number of list items in the cache (cache 3) reaches the preset threshold. If the number of list items in the cache (cache 3) reaches the preset threshold, clear one or more list items in the cache (cache 3); for example, clear the N list items with the shortest cumulative usage time based on the cumulative usage time (totalUseTime) in the list items. If the number of list items in the cache (cache 3) does not reach the preset threshold, the cache management module obtains the text in the current display interface and generates interface hot words based on the obtained text; and creates a list item corresponding to the current display interface in the cache (cache 3), and saves the interface hot words in the list item. In this way, the cache update process is completed, the scene-level hot words corresponding to the current interface (current application) are marked in cache 2, and the interface hot words are saved in cache 3.

[0216] A user can input voice into a terminal device. The terminal device's application interaction module receives the user voice and determines whether the terminal device's display interface has changed between the last time the user voice was received and the current time the user voice was received. If the terminal device's display interface has changed, the voice processing engine retrieves the hotword set (including system-level hotwords, scene-level hotwords, and interface hotwords) corresponding to the current display interface from caches 1, 2, and 3. Optionally, if cache 3 is empty or no list item corresponding to the current display interface exists in cache 3, the application interaction module notifies the cache management module to update the cache. Furthermore, the voice processing engine parses and recognizes the user voice. If the terminal device's display interface has not changed, that is, the current user voice received is not the first time the user voice has been received on the current display interface, the voice processing engine directly parses and recognizes the user voice without retrieving the hotword set. If the recognized content of the user voice successfully matches any hotword in the hotword set, the instruction corresponding to the user voice is executed. Optionally, if the recognized content of the user voice fails to match the hotword set, the cache management module is notified to update the cache.

[0217] It is understandable that, in order to implement the above functions, the above-mentioned terminal device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0218] The embodiment of the present application can divide the functional modules of the terminal device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0219] In the case of using integrated units, a possible structural diagram of the terminal device involved in the above embodiment is shown in FIG14. The terminal device 1400 includes: a processing unit 1401, a storage unit 1402, an audio unit 1403 and a display unit 1404.

[0220] The processing unit 1401 is used to control and manage the actions of the terminal device 1400. For example, it can be used to update the scene-level hot words and interface hot words in the cache when switching applications or switching interfaces within applications; it can be used to parse and recognize user voice when receiving user voice, and / or be used for other processes of the technology described herein.

[0221] The storage unit 1402 is used to store program codes and data of the terminal device 1400. For example, it stores system-level hot words, scene-level hot words, interface hot words, etc. The processing unit 1401 calls the program code stored in the storage unit 1402 to execute each step in the above method embodiment.

[0222] The audio unit 1403 is used to collect the voice of the user and play the voice.

[0223] The display unit 1404 is used to display the interface of the terminal device 1400 .

[0224] Of course, the unit modules in the terminal device 1400 include, but are not limited to, the processing unit 1401, storage unit 1402, audio unit 1403, and display unit 1404. For example, the terminal device 1400 may also include a power supply unit, a communication unit, etc. The power supply unit is used to supply power to the terminal device 1400. The communication unit is used to support communication between the terminal device 1400 and other devices.

[0225] The processing unit 1401 may be a processor or controller, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may include an application processor and a baseband processor. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The storage unit 1402 may be a memory. The audio unit 1403 may include a microphone, a speaker, a receiver, etc. The display unit 1404 may be a display screen, etc. The communication unit may be a transceiver, a transceiver circuit, or a communication interface, etc.

[0226] For example, the processing unit 1401 is a processor (the processor 210 shown in Figure 9), and the storage unit 1402 can be a memory (the internal memory 221 shown in Figure 9). The audio unit 1403 can include a microphone (the microphone 270C shown in Figure 9), a speaker (the speaker 270A shown in Figure 9), and a receiver (the receiver 270B shown in Figure 9). The display unit 1404 can be a display screen (the display screen 294 shown in Figure 9). The communication unit includes a mobile communication module (the mobile communication module 250 shown in Figure 9) and a wireless communication module (the wireless communication module 260 shown in Figure 9). The mobile communication module and the wireless communication module can be collectively referred to as a communication interface. The terminal device 1400 provided in the embodiment of the present application can be the terminal device 100 shown in Figure 9. Among them, the above-mentioned processor, memory, microphone, display screen and communication interface can be coupled together, for example, via a bus connection. The processor calls the program code stored in the memory to execute the various steps in the above method embodiment.

[0227] An embodiment of the present application also provides a chip system (e.g., a system on a chip (SoC)). As shown in FIG15 , the chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 and the interface circuit 1502 can be interconnected via lines. For example, the interface circuit 1502 can be used to receive signals from other devices (e.g., a memory of a terminal device). For another example, the interface circuit 1502 can be used to send signals to other devices (e.g., the processor 1501 or a touch screen of a terminal device). Exemplarily, the interface circuit 1502 can read instructions stored in the memory and send the instructions to the processor 1501. When the instructions are executed by the processor 1501, the terminal device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete components, which are not specifically limited in the embodiment of the present application.

[0228] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned terminal device, the terminal device executes each function or step in the above-mentioned method embodiment.

[0229] The embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the mobile phone in the above method embodiment.

[0230] Among them, the terminal device 1400, chip system, computer-readable storage medium or computer program product provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0231] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0233] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0235] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0236] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for controlling a terminal device by voice, applied to a terminal device, wherein the terminal device includes a microphone, and is characterized in that: The method comprises: The terminal device displays a first interface, wherein the first interface includes a first switch; In response to the user turning on the first switch, the terminal device turns on the voice control function; the turning on the voice control function includes: turning on the recording channel corresponding to the voice control function, the recording channel being used to obtain the user voice collected by the microphone; Receiving a first voice input by a user when the terminal device displays a second interface, and the terminal device executes a first instruction corresponding to the first voice; wherein the second interface does not include a first text corresponding to the first instruction; In response to the user's operation of turning off the first switch, the terminal device turns off the voice control function; turning off the voice control function includes: turning off the recording channel corresponding to the voice control function.

2. The method according to claim 1, characterized in that The terminal device starts the voice control function including: The terminal device displays a first recording icon and a second recording icon, the first recording icon indicates that the voice control function is turned on, and the second recording icon indicates that the recording channel is turned on.

3. The method according to claim 1 or 2, characterized in that: The first voice matches at least one preset hot word.

4. The method according to claim 3, characterized in that The hot words include at least one of the following: Swipe left, swipe right, swipe up, return to the desktop, return.

5. The method according to claim 3 or 4, characterized in that: The application to which the second interface belongs belongs to the audio category, the short video category, or the long video category, and the hot words include at least one of the following: Play, pause, stop, fast forward, rewind.

6. The method according to claim 3 or 4, characterized in that: The application to which the second interface belongs belongs to the category of electronic book playback, and the hot words include at least one of the following: Previous page, next page, table of contents, next chapter.

7. The method according to any one of claims 3 to 6, characterized in that: Before receiving the first voice input by the user when the terminal device displays the second interface, the method further includes: When it is detected that the second interface is switched to the foreground display, the terminal device updates the hot words corresponding to the second interface in the cache; After receiving the first voice input by the user when the terminal device displays the second interface, the method further includes: The terminal device obtains the hot word corresponding to the second interface from the cache.

8. The method according to claim 7, characterized in that The terminal device updates the hot words corresponding to the second interface in the cache, including: The terminal device obtains a scene-level hot word corresponding to the second interface; the scene-level hot word is applicable to all applications in a scene; wherein the scene is pre-configured according to the type of application, and the correspondence between the scene and the scene-level hot word is pre-configured; The terminal device marks the scene-level hot words corresponding to the second interface in the cache.

9. The method according to claim 8, characterized in that The cache of the terminal device includes at least one list item, each list item is used to store an interface hot word corresponding to an interface, and the interface hot word is applicable to an interface. The terminal device updates the hot words corresponding to the second interface in the cache, including: If the list item corresponding to the second interface does not exist in the cache of the terminal device, the terminal device creates a list item corresponding to the second interface in the cache; the list item corresponding to the second interface is used to store the interface hot words corresponding to the second interface.

10. The method according to claim 9, characterized in that After the terminal device creates a list item corresponding to the second interface in the cache, the method further includes: The terminal device acquires text included in the second interface; The terminal device saves the interface hot words corresponding to the second interface in the list item corresponding to the second interface according to the text included in the second interface.

11. The method according to claim 10, characterized in that Before the terminal device creates a list item corresponding to the second interface in the cache, the method further includes: If the number of list items in the cache is greater than or equal to a preset threshold, one or more list items in the cache are cleared.

12. The method according to claim 11, characterized in that The clearing one or more list items in the cache includes: Clear the list items corresponding to one or more interfaces in the cache that have the shortest total cumulative calling time.

13. The method according to any one of claims 7 to 12, characterized in that: The first voice is the user voice received for the first time after the second interface switches to the foreground display.

14. The method according to claim 9, characterized in that The terminal device obtains the hot word corresponding to the second interface from the cache, including: The terminal device obtains system-level hot words, scene-level hot words and interface hot words corresponding to the second interface from the cache; the system-level hot words are pre-configured and applicable to any application on the terminal device.

15. The method according to claim 14, characterized in that The method further comprises: The terminal device obtains the parsed text of the first speech; The terminal device performs voice recognition on the parsed text of the first voice according to the hot words corresponding to the second interface.

16. The method according to claim 15, characterized in that The terminal device performs speech recognition on the parsed text of the first speech according to the hot words corresponding to the second interface, including: The terminal device matches the parsed text of the first voice with the system-level hotword; If the parsed text of the first voice fails to match the system-level hotword, matching the parsed text of the first voice with the scene-level hotword corresponding to the second interface; If the parsed text of the first voice fails to match the scene-level hot words corresponding to the second interface, the parsed text of the first voice is matched with the interface hot words corresponding to the second interface.

17. A terminal device, characterized in that: The terminal device includes: a processor, a memory, a microphone and a display screen, the microphone is used to collect user voice, the display screen is used to display the interface of the terminal device, the processor is coupled to the memory; the memory is used to store computer program code; the computer program code includes computer instructions, and when the processor executes the above-mentioned computer instructions, the terminal device executes the method described in any one of claims 1-16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 16.

Images