Method for controlling terminal equipment through voice and terminal equipment

Abstract

The invention provides a method for controlling terminal equipment through voice and the terminal equipment, and relates to the technical field of terminals. And the terminal equipment starts a voice control function in response to the operation of turning on the switch by the user. After the voice control function is started, the first instruction corresponding to the first voice input by the user can be executed. Wherein an interface displayed by the terminal equipment does not comprise a text corresponding to the first instruction. In other words, the voice instruction supported by the terminal device is not limited to the text in the interface of the terminal device. The terminal device can support a user to control the terminal device through more, more comprehensive and richer voices.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method for controlling a terminal device through voice and a terminal device. Background Art

[0002] With the development of terminal technology, terminal devices are increasingly used in people's daily lives. In order to improve the operating efficiency of users using terminal devices, terminal devices generally provide a variety of human-computer interaction methods such as human-computer interaction interface, voice control, gesture control, etc. Among them, voice control is a major and quick human-computer interaction method. In particular, when users are driving, cooking, reading, and other scenarios where their hands are occupied, it is very convenient and quick to control the terminal device by voice.

[0003] Due to the needs of user information security, terminal device power consumption and other aspects, the voice control function needs to be turned on before it can be used. In some designs, the voice control function is turned on by a wake-up word. The user inputs the wake-up word to the terminal device through voice. After the terminal device is awakened, it can perform corresponding operations according to the user's voice. After the user stops voice input (such as pausing for a few seconds), the voice control function is automatically turned off. Each time the user uses the voice control function, he needs to wake up the terminal device by voice inputting the wake-up word. If you need to control the terminal device by voice for a long time, you have to wake up the terminal device frequently, which is cumbersome.

[0004] In some designs, the voice control function is turned on or off by setting a switch. The user can turn on the voice control function by turning on the preset switch on the terminal device, or turn off the preset switch on the terminal device to turn off the voice control function. While the switch is on, the terminal device can execute corresponding instructions according to the voice input by the user, and the user can easily control the terminal device by voice. At present, the terminal device supports a relatively small range of voices in this scenario. For example, it can only support voices that match the text in the display interface; the user experience of using the voice control function is poor. Summary of the invention

[0005] The embodiments of the present application provide a method and a terminal device for controlling a terminal device by voice. The user does not need to input a wake-up word. As long as a preset switch is turned on, the user can perform human-computer interaction with the terminal device by voice at any time. The terminal device supports richer and more comprehensive voices, thereby improving the user experience of controlling the terminal device by voice.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a method for controlling a terminal device by voice is provided, which is applied to a terminal device, the terminal device including a microphone, and the method comprising: the terminal device displays a first interface, the first interface including a first switch; in response to a user's operation of turning on the first switch, the terminal device turns on a voice control function. In response to a user's operation of turning off the first switch, the terminal device turns off the voice control function. The voice control function is turned on or off by the first switch, and there is no need to wake up the terminal device. For example, the voice control function is a visible-to-speak function. After the voice control function is turned on, a first voice input by a user when the terminal device displays a second interface is received, and the terminal device executes a first instruction corresponding to the first voice; wherein the second interface does not include a first text corresponding to the first instruction. That is, the voice instructions supported by the terminal device are not limited to the text in the interface. The terminal device can support users to control the terminal device through more, more comprehensive, and richer voices.

[0008] Among them, turning on the voice control function includes: turning on the recording channel corresponding to the voice control function, and the recording channel is used to obtain the user's voice collected by the microphone; turning off the voice control function includes: turning off the recording channel corresponding to the voice control function. After the recording channel corresponding to the voice control function is turned on, the voice control function can obtain the voice (audio stream) collected by the microphone through the recording channel.

[0009] In combination with the first aspect, in a possible implementation, the terminal device turning on the voice control function includes: the terminal device displays a first recording icon and a second recording icon. The first recording icon indicates that the voice control function is turned on, so that the user can easily obtain the status of the voice control function being turned on or off through the recording icon. The second recording icon indicates that the recording channel is turned on, which is used to prompt the user that the terminal device is in a voice collection state, thereby preventing the user's privacy from being leaked.

[0010] In combination with the first aspect, in a possible implementation, the first voice matches at least one preset hot word.

[0011] In this method, some hot words are pre-set in the terminal device, and these hot words are not obtained from the display interface. If the voice input by the user matches any hot word, the instruction corresponding to the voice can be executed. Since the hot words are not obtained from the display interface, the voice instructions supported by the terminal device are not limited to the text in the interface. In this way, the terminal device can support more comprehensive and richer voice instructions.

[0012] In some implementations, the pre-set hot words include at least one of the following: left swipe, right swipe, up swipe, back to desktop, and return. These hot words are system-level hot words and are applicable to any application on the terminal device.

[0013] In some implementations, the application to which the second interface belongs belongs to the audio category, the short video category, or the long video category, and the pre-set hot words include at least one of the following: play, pause, stop, fast forward, and fast rewind.

[0014] In some implementations, the application to which the second interface belongs belongs to the category of electronic book playback, and the pre-set hot words include at least one of the following: previous page, next page, directory, and next chapter.

[0015] In combination with the first aspect, in a possible implementation, before receiving the first voice input by the user when the terminal device displays the second interface, it is detected that the second interface is switched to the foreground display, and the terminal device updates the hot words corresponding to the second interface in the cache; after receiving the first voice input by the user when the terminal device displays the second interface, the terminal device obtains the hot words corresponding to the second interface from the cache.

[0016] In this method, when the interface switches to the foreground, the corresponding hot words in the cache are updated. When the user's voice is received, the hot words corresponding to the interface are directly obtained from the cache, without the need for real-time analysis from the current display interface. The voice recognition rate is faster, and the terminal device responds to the user's voice faster, which improves the user experience.

[0017] In combination with the first aspect, in a possible implementation, the terminal device updates the hot words corresponding to the second interface in the cache, including: the terminal device obtains the scene-level hot words corresponding to the second interface, and marks the scene-level hot words corresponding to the second interface in the cache. The scene is pre-configured according to the type of application, and the correspondence between the scene and the scene-level hot words is pre-configured.

[0018] In combination with the first aspect, in a possible implementation, the cache of the terminal device includes at least one list item, and each list item is used to store an interface hot word corresponding to an interface (the interface hot word is applicable to an interface). The terminal device updates the hot word corresponding to the second interface in the cache, including: if the list item corresponding to the second interface does not exist in the cache of the terminal device, the terminal device creates a list item corresponding to the second interface in the cache; the list item corresponding to the second interface is used to store the interface hot word corresponding to the second interface. Then, the terminal device obtains the text included in the second interface; the terminal device saves the interface hot word corresponding to the second interface in the list item corresponding to the second interface according to the text included in the second interface.

[0019] In this method, the interface hot words corresponding to each interface are stored in the cache. The cache may include multiple list items, each of which stores an interface hot word corresponding to an interface. In this way, when the display interface is frequently switched back and forth, since the interface hot words corresponding to multiple display interfaces are stored in the cache, it is not necessary to frequently parse the text from the display interface again when the display interface is switched, which reduces the power consumption generated by parsing the text and improves the response speed of the terminal device.

[0020] In combination with the first aspect, in a possible implementation, before the terminal device creates a list item corresponding to the second interface in the cache, if the number of list items in the cache is greater than or equal to a preset threshold, one or more list items in the cache are cleared.

[0021] In this method, the number of list items stored in the cache is less than or equal to a preset threshold, which can save storage space and avoid excessive cache size caused by linked list anomalies.

[0022] In combination with the first aspect, in a possible implementation, clearing one or more list items in the cache includes: clearing list items corresponding to one or more interfaces in the cache that have the shortest total cumulative calling time.

[0023] In combination with the first aspect, in a possible implementation, the first voice is the user voice received for the first time after the second interface is switched to the foreground. That is to say, it is only necessary to obtain the hot words corresponding to the interface from the cache when the user voice is received for the first time after the interface is switched to the foreground; if the user voice is not received for the first time after the interface is switched to the foreground, the voice processing engine can directly parse and recognize the user voice without obtaining the hot word set again.

[0024] In this method, since the hot words correspond to the interfaces, if the interface is not switched, there is no need to re-acquire the hot words. The processing flow is more concise.

[0025] In combination with the first aspect, in a possible implementation, when the second interface switches to the foreground, the terminal device obtains system-level hot words, scene-level hot words corresponding to the second interface, and interface hot words from the cache after receiving the user's first voice for the first time.

[0026] The terminal device obtains the parsed text of the first voice; and performs voice recognition on the parsed text of the first voice according to the hot words corresponding to the second interface. The terminal device first matches the parsed text of the first voice with the system-level hot words; if the parsed text of the first voice fails to match the system-level hot words, the parsed text of the first voice is matched with the scene-level hot words corresponding to the second interface; if the parsed text of the first voice fails to match the scene-level hot words corresponding to the second interface, the parsed text of the first voice is matched with the interface hot words corresponding to the second interface.

[0027] The priority of system-level hot words is higher than that of scene-level hot words, and the priority of scene-level hot words is higher than that of interface hot words. In this way, speech analysis and recognition can be closer to the user's real intention and avoid misoperation.

[0028] In a second aspect, a terminal device is provided, which has the function of implementing the method described in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0029] In a third aspect, a terminal device is provided, comprising: a processor, a memory and a microphone; the memory is used to store computer execution instructions, and when the terminal device is running, the processor executes the computer execution instructions stored in the memory to enable the terminal device to perform a method as described in any one of the above-mentioned first aspects.

[0030] In a fourth aspect, a terminal device is provided, comprising: a processor; the processor is used to couple with a memory, and after reading instructions in the memory, execute a method as described in any one of the above-mentioned first aspects according to the instructions.

[0031] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computer, the computer can execute any of the methods described in the first aspect.

[0032] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect.

[0033] In a seventh aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting a terminal device to implement the functions involved in the first aspect. In one possible design, the device also includes a memory for storing program instructions and data necessary for the terminal device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0034] Among them, the technical effects brought about by any design method in the second to seventh aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1A A schematic diagram of a scenario example of a method for controlling a terminal device by voice;

[0036] Figure 1B A schematic diagram of another scenario example of a method for controlling a terminal device by voice;

[0037] Figure 2A Schematic diagram 1 of another scenario example of a method for controlling a terminal device by voice;

[0038] Figure 2B Schematic diagram 2 of another scenario example of a method for controlling a terminal device by voice;

[0039] Figure 3 A flow chart of a method for controlling a terminal device by voice;

[0040] Figure 4A Schematic diagram 1 of a scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0041] Figure 4B Schematic diagram 2 of a scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0042] Figure 5 A scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application Figure 3 ;

[0043] Figure 6 Schematic diagram 4 of a scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0044] Figure 7 A scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application Figure 5 ;

[0045] Figure 8 A scenario example of a method for controlling a terminal device by voice provided in an embodiment of the present application Figure 6 ;

[0046] Fig. 9 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0047] Fig.10 A schematic diagram of the software architecture of a terminal device provided in an embodiment of the present application;

[0048] Fig.11 Schematic diagram 1 of the interaction of various modules of a terminal device when implementing the method for controlling a terminal device by voice provided in an embodiment of the present application;

[0049] Fig.12 A second schematic diagram of the interaction of various modules of a terminal device when implementing the method for controlling a terminal device by voice provided in an embodiment of the present application;

[0050] Fig.13A flowchart of a method for controlling a terminal device by voice provided in an embodiment of the present application;

[0051] Fig.14 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0052] Fig.15 A schematic diagram of the chip system structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0054] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include such expressions as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0055] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0056] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] Voice control function:

[0058] Many terminal devices support voice control. The terminal device collects user voice through a microphone, analyzes and recognizes the user voice, and executes the instructions corresponding to the user voice, so that the user can control the terminal device through voice.

[0059] Generally speaking, in order to save power consumption of terminal devices and avoid false triggering, the voice control function needs to be turned on before it can be used. For example, the user inputs a preset word (called a wake-up word) to the terminal device through voice to wake up the terminal device. After the terminal device is awakened, it can execute the instructions corresponding to the user's voice, that is, the voice control function is turned on. For example, the user can turn on or off the voice control function by turning on or off the preset switch in the human-computer interaction interface of the terminal device.

[0060] In different terminal devices, the voice control function may have different names, such as "voice control", "intelligent voice", "voice assistant", "see and speak", "voice command", "free command", "intelligent AI", etc. The specific implementation of voice control functions with different names may also be different.

[0061] Several different implementations of the voice control function are described below by way of example.

[0062] Voice Assistant:

[0063] Before the user uses the voice assistant to control the terminal device, the terminal device needs to be woken up first. In one example, the wake-up word input by the user to the terminal device is detected, and the terminal device is awakened. In another example, the terminal device is awakened by detecting the user's long press of the power button. In another example, the terminal device is awakened by detecting the breath generated when the user inputs voice to the terminal device. Generally speaking, before the terminal device is awakened, the microphone of the terminal device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone is only detected for wake-up words at the kernel layer, and the corresponding recording channel of the voice assistant is not started in the system and driver of the terminal device.

[0064] In response to user operations (such as receiving a wake-up word input by the user's voice), the terminal device is awakened, and the corresponding recording channel of the voice assistant is started in the system and driver. After the terminal device is awakened, the voice (audio stream) collected by the microphone is sent to the voice assistant application for processing through the recording channel corresponding to the voice assistant. In this way, the terminal device can execute the instructions corresponding to the user's voice, enabling the user to control the terminal device through voice; it can also realize functions such as dialogue with the user.

[0065] For example, Figure 1A The figure shows a schematic diagram of a scenario in which a user uses a voice assistant to control a mobile phone. Figure 1A As shown, the mobile phone displays the desktop interface, and the user inputs the voice "Hello YOYO" into the mobile phone. In response to receiving the wake-up word "Hello YOYO", the mobile phone is awakened. For example, after the mobile phone is awakened, the voice "I am here" is played to remind the user that the mobile phone has been awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. For example, as Figure 1A As shown, the user inputs the voice "open video" to the mobile phone, and the mobile phone analyzes and recognizes the user's voice and executes the command corresponding to the voice "open video". Exemplarily, the mobile phone starts the video application in response to receiving the user's voice "open video".

[0066] For example, Figure 1B FIG. 1 shows another example of a scenario in which a user uses a voice assistant to control a mobile phone. Figure 1B As shown, the mobile phone displays the desktop interface, and the user long presses the power button. In response to receiving the user's long press of the power button operation, the mobile phone is awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. For example, Figure 1B As shown, the user inputs the voice "open video" to the mobile phone, and the mobile phone analyzes and recognizes the user's voice and executes the command corresponding to the voice "open video". Exemplarily, the mobile phone starts the video application in response to receiving the user's voice "open video".

[0067] In some implementations, after the terminal device is awakened, the user can issue a command to the terminal device by inputting voice into the terminal device, and the terminal device executes the command corresponding to the voice. After the terminal device executes a command, or if the terminal device does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after being awakened, the terminal device no longer responds to the command issued by the user through voice. For example, the terminal device will close the recording channel corresponding to the voice assistant. The user needs to input the wake-up word to the terminal device again to wake up the terminal device before he can issue a command to the terminal device by inputting voice into the terminal device again. In other words, after the terminal device is awakened, it enters a "short reception" state and can respond to the user's voice command within a short period of time (for example, within 8 seconds).

[0068] In some implementations, when the terminal device is connected to the network, it supports entering a continuous conversation scenario after being awakened, and the user can have a continuous conversation with the terminal device. After each broadcast of the terminal device is completed, it will continue to pick up the sound without repeated awakening. Until the user exits the continuous conversation through instructions such as "exit".

[0069] It can be seen that:

[0070] It can be said that it is implemented locally by the terminal device and does not need to be connected to the network.

[0071] Visible and Talkable is controlled by a preset switch. Users can turn on Visible and Talkable by turning on the preset switch, and can stop Visible and Talkable by turning off the preset switch.

[0072] Taking the terminal device as a mobile phone as an example, for example, Figure 2A As shown, the user can open the settings function of the mobile phone 100; for example, the user clicks on the application icon of the "Settings" application on the desktop. In response to the user clicking on the application icon of the "Settings" application, the mobile phone 100 displays the "Settings" interface 101. The "Settings" interface 101 includes a "Smart Voice" option 102, and the "Smart Voice" option 102 is used to set the smart voice function. Exemplarily, in response to the user clicking on the "Smart Voice" option 102, the mobile phone 100 displays the "Smart Voice" interface 103, and the "Smart Voice" interface 103 includes a "Visible and Speakable" option 104. The user can click on the "Visible and Speakable" option 104 to set options related to the Visible and Speakable function. Exemplarily, refer to Figure 2A In response to the user's click operation on the "See and Say" option 104, the mobile phone 100 displays the "See and Say" interface 105. Optionally, the "See and Say" interface 105 includes a prompt message 106 for prompting the user how to use the "See and Say" function. The "See and Say" interface 105 also includes a "See and Say" switch 107. The user can click the "See and Say" switch 107 to turn on or off the "See and Say" switch. In one example, in response to receiving the user's click operation on the "See and Say" switch 107, the "See and Say" switch of the mobile phone 100 is turned on to turn on the "See and Say" function. Optionally, the "See and Say" interface 105 displays a prompt message 108 for prompting the user that the "See and Say" function has been successfully turned on.

[0073] In one implementation, after the see-and-speak function is turned on, the mobile phone 100 displays a first recording icon, which indicates that the see-and-speak function is turned on. Figure 2A As shown, after the "visible and speakable" switch 107 is turned on, the status bar of the display interface of the mobile phone 100 displays a recording icon 10a, indicating that the visible and speakable function has been turned on.

[0074] In one scenario, the preset switch corresponding to Visible and Talk on the terminal device is not turned on, and the microphone of the terminal device is not turned on. When the preset switch corresponding to Visible and Talk is turned on, the terminal device starts the microphone and starts the corresponding recording channel of Visible and Talk in the system and driver. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talk application for processing through the corresponding recording channel of Visible and Talk, so that the user can control the terminal device through voice.

[0075] In another scenario, the preset switch corresponding to Visible and Talking of the terminal device is not turned on, and the microphone of the terminal device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. When the preset switch corresponding to Visible and Talking is turned on, the corresponding recording channel of Visible and Talking is started in the system and driver of the terminal device. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talking application for processing through the corresponding recording channel of Visible and Talking, so that the user can control the terminal device through voice.

[0076] After Visible is Talking is turned on, the corresponding recording channel of Visible is Talking is started in the system and driver of the terminal device, and the terminal device enters a "long receiving" state to continuously collect surrounding sounds. Users can give commands to the terminal device at any time through voice without having to enter a wake-up word to wake up the terminal device.

[0077] In one embodiment, after any function on the terminal device turns on the voice recording function of the terminal device (turns on the microphone and turns on the recording channel), the terminal device will send a prompt message to the user to remind the user that the terminal device is in the voice collection state. In this way, the user's privacy can be prevented from being leaked. For example, after the visible and visible function is turned on, the terminal device enters a continuous voice collection state, and a second recording icon is displayed on the terminal device display interface. The second recording icon indicates that the recording channel is turned on, which is used to remind the user that the microphone is collecting voice. For example, Figure 2A As shown, the status bar of the display interface of the mobile phone 100 displays a recording icon 10b, indicating that the recording channel is turned on.

[0078] After the "See and Speak" function is turned on, the terminal device starts the recording channel and continuously collects the user's voice through the microphone. The user can input voice into the terminal device at any time. The terminal device analyzes and recognizes the user's input voice and executes the instructions corresponding to the user's voice.

[0079] In one example, if Figure 2BAs shown, the mobile phone 100 displays a desktop interface. The user inputs a voice message “open video” to the mobile phone 100. In response to receiving the voice message “open video” input by the user, the mobile phone 100 starts the video application and displays the user interface 120 of the video application.

[0080] Hot words:

[0081] The terminal device can be set with some hot words, that is, after it is turned on, if the received user voice matches at least one of the set hot words, the terminal device executes the instruction corresponding to the user voice.

[0082] The hot words may be pre-configured on the terminal device, or may be obtained based on the content in the display interface of the terminal device, or may be input by the user, etc.

[0083] In one implementation, each time an application switch or an in-application display interface switch is detected, the terminal device obtains text information of the current interface of the foreground application and sets a hot word based on the obtained text information. In this way, the user can input a voice matching the hot word to control the terminal device to execute the corresponding instruction.

[0084] Exemplary, reference Figure 3 , the user turns on the switch of "See and Speak" on the terminal device, and turns on "See and Speak". After that, application 1 switches to the foreground to run. The voice control module obtains the elements (controls, etc.) in the display interface of application 1, obtains the interface information (the text included in the display interface), and refreshes the saved interface information. After that, application 2 switches to the foreground to run. The voice control module obtains the elements (controls, etc.) in the display interface of application 2, and refreshes the saved interface information. The user inputs voice to the terminal device, and the voice control module sets hot words according to the saved interface information. If it is determined that the received user voice matches at least one hot word, the instruction corresponding to the user voice is executed.

[0085] Exemplary, reference Figure 2BIn the scenario shown, the mobile phone displays the desktop interface. After detecting that the desktop application has switched to the foreground, the mobile phone obtains and saves the text information in the desktop interface, including "5G", "8:00", "Clock", "Calendar", "Gallery", "Memo", "File Management", "Email", "Music", "Calculator", "App Store", "Sports Health", "Weather", "Browser", "Settings", "AR Application", "Video", "Camera", "Contacts", "Phone", "Information", etc. The text information "5G", "8:00", "Clock", "Calendar", "Gallery", "Memo", "File Management", "Email", "Music", "Calculator", "App Store", "Sports Health", "Weather", "Browser", "Settings", "AR Application", "Video", "Camera", "Contacts", "Phone", "Information", etc. are set as hot words.

[0086] The user inputs the voice "open video" into the mobile phone. After receiving the user's voice, the mobile phone analyzes and recognizes the user's voice, determines that the voice "open video" contains the hot word "video", that is, determines that the voice "open video" matches the hot word "video", and the mobile phone executes the instruction corresponding to the voice "open video". For example, Figure 2B As shown, in response to receiving the voice input of "open video" by the user, the mobile phone starts the video application and displays the user interface of the video application.

[0087] The video application switches to the foreground, and the phone obtains and saves the text information in the video application display interface, including "5G", "8:00", "media title", "selection", "1", "2", "3", "4", "5", "6", "7", "8", "9", etc. "5G", "8:00", "media title", "selection", "1", "2", "3", "4", "5", "6", "7", "8", "9", etc. are set as hot words. The user inputs voice into the phone. If the user's voice matches any hot word, the phone can execute the corresponding command.

[0088] In this implementation, each time an application is switched or the display interface in the application is switched, the terminal device will update the hot words according to the text information of the current display interface. The hot words are set and updated according to the text in the current display interface of the terminal device. Text not included in the display interface will not be set as a hot word. The acquisition scenario of hot words is relatively limited. In this way, the voices supported by the terminal device are relatively limited and the types are relatively small. For example, the mobile phone display Figure 2B When the video application displays the interface, if the user inputs the voice "pause", the phone confirms that the voice "pause" does not match any hot words and will not execute the corresponding command.

[0089] The embodiment of the present application provides a method for controlling a terminal device through voice, which not only supports setting text in the display interface as a hot word, but also supports setting text not included in the display interface as a hot word; the acquisition of hot words does not depend on the text in the display interface. In this way, after the visible is spoken function is turned on, the terminal device can support users to control the terminal device through more, more comprehensive, and richer voice.

[0090] After the "See and Speak" function is turned on, the terminal device enters a continuous sound receiving state. The user can input voice into the terminal device. The terminal device analyzes and recognizes the voice input by the user. If the user's voice is recognized to match any hot word, the command corresponding to the voice is executed.

[0091] Exemplary, reference Figure 4A , the mobile phone 100 displays the short video playback interface 121 of the short video application, and receives the voice input of the user "return to the desktop". The text information in the short video playback interface 121 includes "5G", "8:00", "Follow", "Recommend", "Nearby", "Home", "Friends", "Message", "Me", etc., but does not include "return to the desktop". In the method of controlling the terminal device by voice provided in the embodiment of the present application, the acquisition of hot words does not depend on the text in the display interface, and "return to the desktop" can be set as a hot word. The mobile phone 100 recognizes that the user's voice "return to the desktop" matches the hot word "return to the desktop", and executes the instruction corresponding to the voice "return to the desktop". In one example, in response to receiving the voice input of the user "return to the desktop", the desktop application of the mobile phone 100 switches to the foreground to run, and the mobile phone 100 displays the desktop interface. In this example, the hot word "return to the desktop" is not obtained from the mobile phone display interface, and the acquisition of the hot word does not depend on the text in the display interface. The hot word can be pre-configured on the mobile phone.

[0092] In the method for controlling a terminal device by voice provided in an embodiment of the present application, some hot words are pre-set in the terminal device, and these hot words are not obtained from the display interface. In one implementation, the terminal device also obtains some hot words from the display interface. That is, on the terminal device, some hot words come from the text in the display interface, and the other hot words are pre-set. In this way, the terminal device supports more and more comprehensive hot words; accordingly, the terminal device can support users to control the terminal device through more, more comprehensive, and richer voices.

[0093] In some embodiments, system-level hot words are pre-set in the terminal device, and the system-level hot words are applicable to all applications on the terminal device. When any application on the terminal device is running in the foreground, if the voice input by the user matches at least one system-level hot word, the terminal device executes the instruction corresponding to the user's voice. System-level hot words are global and do not depend on applications or application interfaces. In one example, system-level hot words may include: swipe left, swipe right, swipe up, return to desktop, return, etc.

[0094] Exemplary, reference Figure 4A When the mobile phone 100 displays the interface of the short video application, the mobile phone 100 can respond to the user's voice input "return to the desktop" and execute the return to the desktop command.

[0095] In another example, Figure 4B As shown, the mobile phone 100 displays the application interface 122 of the e-book application and receives the voice input of the user "return to the desktop". The mobile phone 100 recognizes that the user's voice "return to the desktop" matches the hot word "return to the desktop", and then executes the instruction corresponding to the voice "return to the desktop". For example, Figure 4B As shown, the desktop application of the mobile phone 100 is switched to the foreground and the mobile phone 100 displays the desktop interface.

[0096] It should be noted that the mobile phone 100 can understand the semantics of the user's voice. That is, the user's voice may not be completely consistent with the hot word, as long as the semantics are the same or close. That is, the user's voice and the hot word match, but they do not have to be consistent. For example, in Figure 4B In the scenario shown, the user inputs the voice "return to desktop", and the system-level hot words include "return to desktop". "Return to desktop" and "return to desktop" have the same semantics, that is, the two match.

[0097] In another example, Figure 5 As shown, the mobile phone 100 displays the desktop interface and receives the user's voice input "swipe left". The mobile phone 100 recognizes that the user's voice "swipe left" matches the system-level hot word "swipe left", and then executes the instruction corresponding to the voice "swipe left". For example, Figure 5 As shown, the mobile phone 100 displays a negative one screen interface.

[0098] In some embodiments, scene-level hot words are pre-set in the terminal device, and the scene-level hot words are applicable to all applications in a scene. In one implementation, multiple scenes are pre-set in the terminal device, and each scene corresponds to at least one application. Exemplarily, the pre-set scenes may include audio and video scenes, e-book scenes, home screen scenes, permission pop-up scenes, etc.

[0099] In one example, the correspondence between the application and the scenario is preconfigured in the terminal device. Exemplarily, the correspondence between the application and the scenario is shown in Table 1. The terminal device can determine the scenario corresponding to the application according to the application name (package name of the application).

[0100] Table 1

[0101]

[0102]

[0103] In another example, the correspondence between the application type and the scene is pre-configured in the terminal device. Exemplarily, the correspondence between the application type and the scene is shown in Table 2. The correspondence between the application and the application type may also be pre-configured in the terminal device. Exemplarily, the correspondence between the application and the application type is shown in Table 3. The terminal device may determine the application type according to the application name, and determine the scene corresponding to the application according to the correspondence between the application type and the scene.

[0104] Table 2

[0105] Scenario Application Type Audio and video scenes Audio, short video, long video E-book scene E-book playback Home screen scene Home screen management Permission pop-up scenario Permission Management

[0106] Table 3

[0107]

[0108] It should be noted that the above Tables 1 to 3 are only illustrative of the correspondence between applications and scenarios. In other embodiments, the terminal device may obtain the correspondence between applications and scenarios in other ways; for example, the correspondence between applications and scenarios may not be in the form of a configuration table; for example, the correspondence between applications and scenarios may be different from that in the above examples. In other embodiments, the terminal device may not save the correspondence between applications and scenarios; for example, the terminal device may obtain the correspondence between applications and scenarios from a server. The embodiments of the present application do not limit the specific manner in which the terminal device obtains the scenario to which the application belongs.

[0109] The scene-level hot words are related to the scene to which the application belongs, and are not dependent on the text contained in the application interface. In one example, the scene-level hot words corresponding to each scene are shown in Table 4.

[0110] Table 4

[0111] Scenario Scene-level hot words Audio and video scenes Play, Pause, Stop, Fast Forward, Fast Rewind E-book scene Previous page, Next page, Table of contents, Next chapter Home screen scene Application list information, setting menu items Permission pop-up scenario Allow, confirm, deny, I understand

[0112] It is understandable that in other implementations, the way of dividing scenes may be different from the above examples; for example, more or fewer scenes may be divided. The scene-level hot words corresponding to each scene may also be adjusted according to actual conditions.

[0113] When any application on the terminal device is running in the foreground, in response to receiving a voice input by the user, the terminal device determines the scene to which the application belongs. If the voice input by the user matches at least one scene-level hot word corresponding to the scene to which the application belongs, the terminal device executes the instruction corresponding to the user's voice.

[0114] For example, Figure 6 The diagram shows a scenario example in which a mobile phone receives user voice in an audio and video scenario.

[0115] refer to Figure 6 , the mobile phone 100 displays a short video application (for example, ) of the short video playback interface 121. The scene to which the short video application belongs is the audio and video scene, and the scene-level hot words corresponding to the audio and video scene include: play, pause, stop, fast forward, fast rewind, etc. The user inputs the voice "pause" to the mobile phone 100. The voice "pause" input by the user is received, and the voice "pause" matches the hot word "pause". In response to receiving the voice "pause", the mobile phone 100 executes the instruction corresponding to the voice "pause" to pause the short video. In one example, Figure 6 As shown, a "play" button 123 is displayed on the short video playback interface 121 to prompt the user that the short video has been paused. The user can click the "play" button 123 to continue playing the short video. The user can also input a voice "play" into the mobile phone 100 to continue playing the short video.

[0116] For example, Figure 7 The diagram shows a scenario example in which a mobile phone receives user voice in an e-book scenario.

[0117] refer to Figure 7 , the mobile phone 100 displays an e-book application (e.g., ) application interface 122. The scene to which the e-book application belongs is the e-book scene, and the scene-level hot words corresponding to the e-book scene include: previous page, next page, directory, next chapter, etc. The user inputs the voice "next page" to the mobile phone 100. The voice "next page" input by the user is received, and the voice "next page" matches the hot word "next page". In response to receiving the voice "next page", the mobile phone 100 executes the instruction corresponding to the voice "next page" and displays the e-book application (for example, ) of the application interface 124. The page number of the application interface 124 is page 20, the page number of the application interface 122 is page 19, and the application interface 124 is the next page of the application interface 122.

[0118] In some implementations, the terminal device also obtains hot words from the display interface of the foreground application, that is, obtains interface hot words. Figure 8 , the mobile phone 100 displays a short video application (for example, ) of the short video playback interface 121. The text information in the short video playback interface 121 includes "5G", "8:00", "Follow", "Recommend", "Nearby", "Home", "Friends", "Message", "Me", etc. The mobile phone 100 sets the text in the current display interface of the foreground application as the interface hot words, that is, the interface hot words include "5G", "8:00", "Follow", "Recommend", "Nearby", "Home", "Friends", "Message", "Me", etc.

[0119] The mobile phone 100 receives the voice "check message" input by the user, and the voice "check message" matches the interface hot word "message", and the mobile phone 100 executes the instruction corresponding to the voice "check message". For example, Figure 8 As shown, the mobile phone 100 displays a message interface 125 .

[0120] The method for controlling a terminal device through voice provided in an embodiment of the present application, the terminal device supports pre-set system-level hot words, scene-level hot words, and interface hot words obtained from the foreground application display interface, and supports more and more comprehensive types of hot words. Accordingly, it can support users to control the terminal device through more, more comprehensive and richer voices.

[0121] The method for controlling a terminal device by voice provided in the embodiment of the present application can be applied to a terminal device that supports voice input. The above-mentioned terminal devices may include mobile phones, tablet computers, laptop computers, personal computers (PC), ultra-mobile personal computers (UMPC), handheld computers, netbooks, smart home devices (e.g., smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (e.g., smart watches, smart bracelets, etc.), vehicle-mounted devices, virtual reality devices, etc., and the embodiments of the present application do not impose any restrictions on this.

[0122] For example, Fig. 91 is a schematic diagram of the structure of the terminal device 100. The terminal device 100 may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295. The sensor module 280 may include a temperature sensor, an ambient light sensor, a touch sensor, and the like.

[0123] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device 100. In other embodiments, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0124] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0125] The controller may be the nerve center and command center of the terminal device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0126] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0127] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0128] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation on the terminal device. In other embodiments, the terminal device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0129] The charging management module 240 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 may receive charging input from a wired charger through the USB interface 230. In some wireless charging embodiments, the charging management module 240 may receive wireless charging input through a wireless charging coil of a terminal device. While the charging management module 240 is charging the battery 242, it may also power the terminal device through the power management module 241.

[0130] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 241 can also be set in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

[0131] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0132] The terminal device 100 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0133] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-OLED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc.

[0134] The terminal device 100 can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0135] The ISP is used to process the data fed back by the camera 293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

[0136] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0137] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0138] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. In this way, the terminal device can play or record videos in multiple coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0139] NPU is a neural network (NN) computing processor. It can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of terminal devices, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0140] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the earphone interface 270D, and the application processor.

[0141] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 are arranged in the processor 210. The speaker 270A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 270B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. The microphone 270C, also known as the "microphone" and the "microphone", is used to convert sound signals into electrical signals. The headphone interface 270D is used to connect wired headphones. The headphone interface 270D can be a USB interface 230, or it can be a 3.5mm open mobile electronic device platform (open mobile terminal platform, OMTP) standard interface, the cellular telecommunications industry association of the USA (cellular telecommunications industry association of the USA, CTIA) standard interface. In the embodiment of the present application, after the voice control function is turned on, the terminal device 100 continuously collects surrounding sounds through the microphone 270C.

[0142] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, audio, video and other files are stored in the external memory card.

[0143] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the terminal device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 can execute instructions stored in the internal memory 221, and the internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store data (such as video files) created during the use of the terminal device, etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0144] The button 290 includes a power button, a volume button, etc. The button 290 can be a mechanical button. It can also be a touch button. The motor 291 can generate a vibration prompt. The motor 291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 292 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 295, or pulled out from the SIM card interface 295 to achieve contact and separation with the terminal device. The terminal device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0145] In the embodiment of the present application, the terminal device 100 is a terminal device that can run an operating system and install application programs. Optionally, the operating system running on the terminal device 100 can be system, system, System, etc.

[0146] In some embodiments, the software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture or a cloud architecture. Taking the system as an example, the software structure of the terminal device 100 is exemplarily described.

[0147] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0148] The application layer may include a series of application packages APK. For example, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications. In the embodiment of the present application, the application layer includes a voice control APK, which is used to provide a voice control function for the terminal device 100. The voice control APK includes an interface monitoring module, a cache management module, an application interaction module, etc. Among them, the interface monitoring module is used to monitor the startup, exit, switching and other processes of the application interface. The cache management module is used to manage the acquisition and caching of hot words. The application interaction module is used to manage the process of the application executing instructions according to voice.

[0149] The application layer also includes a speech processing engine for parsing, recognizing and processing speech. The speech parsing module is used to convert speech into text; the speech recognition module is used to understand and recognize the semantics of the text and determine whether it matches the hot words; the instruction mapping module is used to convert the recognized semantics into machine-executable instructions.

[0150] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0151] like Fig.10 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, an activity manager service (AMS), an application manager (PMS), an interface acquisition and parsing module, a multi-mode control module, and the like.

[0152] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0153] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0154] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0155] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0156] AMS is mainly responsible for the startup, switching, and scheduling of the four major components in the system, as well as the management and scheduling of application processes. Its responsibilities are similar to the process management and scheduling modules in the operating system. When a process or component is started, the request will be passed to AMS through the Binder communication mechanism, and AMS will then handle it uniformly.

[0157] PMS handles package management related tasks, such as application installation, application uninstallation, etc. It is also used to provide all the information of an application, such as startup, shutdown, etc.

[0158] The interface acquisition and parsing module is used to parse elements on the display interface of the terminal device 100, such as controls, text on controls, etc.

[0159] The multi-mode control module is used to manage the execution of voice commands, for example, sending commands to applications so that the applications execute the commands.

[0160] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0161] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0162] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0163] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0164] A 2D graphics engine is a drawing engine for 2D drawings.

[0165] Android runtime is responsible for the scheduling and management of the Android system. Android runtime includes core libraries and virtual machines.

[0166] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0167] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0168] The kernel layer is the layer between hardware and software. The kernel layer can include display drivers, sensor drivers, microphone drivers, Wi-Fi drivers, etc.

[0169] Combination Fig.10 , Fig.11 A schematic diagram of the interaction of various modules of a terminal device when implementing a method for controlling a terminal device by voice provided in an embodiment of the present application is shown.

[0170] The method for controlling a terminal device by voice provided in the embodiment of the present application pre-sets system-level hot words and scene-level hot words for multiple scenes (such as audio and video scenes, e-book scenes, home screen scenes, and permission pop-up scenes). Exemplarily, system-level hot words include: swipe left, swipe right, swipe up, return to desktop, return, etc.; the scene-level hot words corresponding to each scene are shown in Figure 4.

[0171] System-level hot words are global and do not depend on applications or application interfaces. In one implementation, system-level hot words are stored in a terminal device and can be applied to various applications.

[0172] Scenario-level hot words apply to all applications within a scenario. In other words, by determining the scenario to which an application belongs, you can obtain the scenario-level hot words corresponding to the application.

[0173] The terminal device can also obtain the text in the interface according to the display interface of the foreground application, that is, obtain the hot words of the interface.

[0174] That is to say, both the scene-level hot words and the interface hot words are related to the application. When the application is started or switched to the foreground, the terminal device obtains the scene-level hot words and the interface hot words corresponding to the application.

[0175] refer to Fig.11 In some embodiments, in response to a user operation or a call between applications, a terminal device switches between applications or switches the interface within an application. The interface monitoring module of the voice control APK monitors the application switching or the display interface switching within the application.

[0176] In one implementation, Fig.12 As shown, AMS can monitor the life cycle of the Activity corresponding to each interface of each application on the terminal device, including onCreate, onStart, onResume, onPause, onStop, onRestart and onDestroy.

[0177] onCreate: Executed when the Activity is first loaded. The onCreate event is executed when the interface is first started. After the Activity is destroyed, the onCreate event is re-executed after it is reloaded.

[0178] onStart: Executed after the onCreate event. After the interface is cut out, if you re-enter the interface after a period of time (the Activity is not destroyed), the onCreate event will be skipped and the onStart event will be executed directly.

[0179] onResume: Executed after the onStart event; or after the interface is swapped to the background, when the user views the interface again (the Activity has not been destroyed and the onStop event has not been executed), the onCreate and onStart events will be skipped and the onResume event will be executed directly.

[0180] onPause: Called when the interface is switched to the background.

[0181] onStop: Executed after the onPause event. If the interface is not returned within a period of time, the onStop event of the Activity of the interface will be executed. Or if the user directly removes the interface from the current page, the onStop event of the Activity of the interface will also be executed.

[0182] onRestart: After the onStop event is executed, if the interface and the process of the application are not destroyed by the system, when the user re-enters the interface, the onRestart event of the Activity of the interface will be executed. After the onRestart event, the onCreate event will be skipped and the onStart event will be executed directly.

[0183] onDestroy: Executed when the Activity is destroyed. After executing the onStop event, if the interface is not returned again, the Activity will be destroyed.

[0184] The interface monitoring module registers monitoring with AMS. In this way, when AMS monitors the life cycle changes of Activity, it can call back to notify the interface monitoring module. Among them, the notification includes the component name corresponding to the Activity, and the component name can include the application package name and the Activity name. The interface monitoring module can determine whether an application switch or an in-application display interface switch has occurred based on the life cycle changes of Activity.

[0185] In one example, the display interface 1 of application 1 corresponds to Activity 1, and the display interface 2 corresponds to Activity 2. AMS monitors the onPause event of Activity 1 and the onResume event of Activity 2, and notifies the interface monitoring module of the onPause event of Activity 1 and the onResume event of Activity 2. The interface monitoring module determines that the display interface 1 is switched to the background according to the onPause event of Activity 1, and determines that the display interface 2 is switched to the foreground display according to the onResume event of Activity 2; that is, the display interface switching within the application is detected.

[0186] In another example, the display interface 1 of application 1 corresponds to Activity 1, and the display interface 1 of application 2 corresponds to Activity 2. AMS monitors the onPause event of Activity 1 and the onResume event of Activity 2; it notifies the interface monitoring module of the onPause event of Activity 1 of application 1 and the onResume event of Activity 2 of application 2. The interface monitoring module determines that the display interface 1 of application 1 is switched to the background according to the onPause event of Activity 1 of application 1, and determines that the display interface 1 of application 2 is switched to the foreground display according to the onResume event of Activity 2 of application 2; that is, application switching is detected.

[0187] Continue to refer Fig.11 , the interface monitoring module monitors the application switching or the display interface switching within the application, and notifies the cache management module to refresh the cache of scene-level hot words and the cache of interface hot words; in one implementation, the notification message includes the component name of the display interface switched to the foreground. The cache management module can obtain the application package name and Activity name according to the component name.

[0188] In one example, the cache 1 of the terminal device is used to store preset system-level hot words, and the system-level hot words do not need to be updated.

[0189] Cache 2 is used to save scene-level hot words for each preset scene. The cache management module determines the scene corresponding to the application according to the application package name, and marks the scene-level hot words corresponding to the scene in cache 2. In one implementation, the scene-level hot words for each scene are saved in cache 2 in the form of a list. Exemplarily, List 1 is the scene-level hot words for the audio and video scene, List 2 is the scene-level hot words for the e-book scene, List 3 is the scene-level hot words for the main screen scene, and List 4 is the scene-level hot words for the permission pop-up window scene. The state of each list is inactive by default. The cache management module determines the corresponding scene according to the application package name, and sets the list corresponding to the scene in cache 2 to an active state, and sets the remaining lists to an inactive state. For example, the cache management module determines that the application belongs to the audio and video scene according to the application package name, then sets List 1 to an active state, and sets List 2, List 3 and List 4 to an inactive state. The list set to an active state is the marked list; that is, the scene-level hot words saved in the list are marked. In this way, when the hot word set corresponding to the interface is determined later, the scene-level hot words corresponding to the interface can be easily found from cache 2. It is not necessary to determine the corresponding scene and scene-level hot words in real time according to the application to which the interface belongs each time the hot word set corresponding to the interface is determined, which improves the processing efficiency of the terminal device in obtaining the hot word set after receiving the user's voice.

[0190] Cache 3 is used to store interface hot words. The cache management module updates cache 3 according to the component name (application package name and Activity name) of the display interface switched to the foreground.

[0191] In one implementation, at least one list item may be stored in the cache 3, wherein each list item is used to store a group of interface hot words corresponding to an interface.

[0192] Exemplarily, a list item includes the following parameters:

[0193] lastUseTime (last used time), indicating the time when the interface was last called;

[0194] totalUseTime (accumulated usage time), which indicates the total time the interface has been called;

[0195] cacheId (component identifier), used to uniquely identify a component. For example, cacheId is the component name, including the application package name and Activity name;

[0196] hotword (hot word information), used to save the interface hot words corresponding to this interface.

[0197] In one example, the cache management module matches the component name (application package name and Activity name) corresponding to the display interface switched to the foreground with the cacheId (component identifier) ​​of the list item in cache 3, and searches for the list item corresponding to the display interface switched to the foreground in cache 3.

[0198] If the list item corresponding to the component name does not exist in cache 3, create a list item corresponding to the component name, and update the parameters in the list item accordingly. Exemplarily, lastUseTime = current time, totalUseTime = 0, cacheId = component name corresponding to the display interface switched to the foreground. The cache management module calls the interface acquisition and parsing module to obtain the text in the display interface switched to the foreground. In one implementation, all the text in the acquired display interface is set as an interface hot word. In another implementation, a part of the text parsed from the display interface is set as an interface hot word. For example, in an e-book scenario, the text related to the control included in the text of the display interface can be set as an interface hot word. Furthermore, set hotword in the newly created list item = the interface hot word corresponding to the display interface switched to the foreground.

[0199] If there is a list item corresponding to the component name in cache 3, update the parameter lastUseTime in the list item. Exemplarily, lastUseTime = current time. In addition, obtain the component name of the previous display interface, and search for the corresponding list item in cache 3 according to the component name of the previous display interface. Exemplarily, the cache management module updates the parameter totalUseTime in the list item corresponding to the previous display interface, totalUseTime = current time - lastUseTime. Among them, the previous display interface is the display interface displayed before the terminal device displays the current display interface.

[0200] Multiple list items can be saved in cache 3. When a list item is created, the cacheId and hotword of the list item are recorded; when the display interface switches to the foreground, the lastUseTime in the list item corresponding to the display interface is updated; when the display interface exits the foreground, the totalUseTime in the list item corresponding to the display interface is updated. That is to say, in one implementation, the cache management module only needs to call the interface acquisition and parsing module to obtain the text in the display interface switched to the foreground (i.e., obtain the interface hotword corresponding to the display interface switched to the foreground) when creating a list item of a display interface. It is not necessary to call the interface acquisition and parsing module to parse the display interface and obtain the interface hotword corresponding to the display interface every time the display interface switches to the foreground. The power consumption generated by parsing the display interface when the display interface is frequently switched is reduced.

[0201] In one implementation, the number of list items stored in the cache 3 is less than or equal to a preset threshold, which can save storage space and prevent the cache 3 from being too large due to a linked list anomaly.

[0202] When a new list item needs to be added to cache 3, the number of list items stored in cache 3 is first determined. If the number of list items stored in cache 3 is greater than or equal to a preset threshold, one or more list items stored in cache 3 are cleared so that the number of list items stored in cache 3 is less than the preset threshold; and then the list item is created.

[0203] In one example, one or more list items to be cleared are determined based on totalUseTime in the list items. For example, N list items with the smallest totalUseTime value (ie, the shortest total cumulative calling time of the display interface) are cleared; wherein N is greater than or equal to 1.

[0204] In this method, after monitoring the application switching or the display interface switching within the application, the corresponding scene-level hot words are obtained according to the scene to which the foreground application belongs, and marked in cache 2; the corresponding interface hot words are obtained according to the display interface of the foreground application, and the corresponding list items in cache 3 are updated. In this way, the system-level hot words, scene-level hot words, and interface hot words corresponding to the display interface switched to the foreground can be saved through cache 1, cache 2, and cache 3.

[0205] Furthermore, when the terminal device displays the display interface switched to the foreground, the user can input voice to the terminal device. The microphone of the terminal device receives the user's voice and transmits the user's voice to the application interaction module of the voice control APK. The application interaction module transmits the user's voice to the voice processing engine for processing.

[0206] In one implementation, if the user voice is received for the first time after switching to the current display interface, the voice processing engine calls the hot word acquisition module to update the hot word set. In one example, the hot word acquisition module reads the system-level hot words from cache 1, reads the scene-level hot words corresponding to the scene to which the foreground application belongs from cache 2, and reads the interface hot words corresponding to the current display interface of the foreground application from cache 3. The acquired system-level hot words, scene-level hot words, and interface hot words are merged into a hot word set corresponding to the current display interface of the foreground application, and the hot word set is saved in cache 4. Optionally, in another example, since the system-level hot words are applicable to any application, after the hot word set in cache 4 is generated for the first time, there is no need to update the system-level hot words in the hot word set subsequently. If it is an interface switch within the application, or the application belongs to the same scene before and after the switch, there is no need to update the scene-level hot words in the hot word set.

[0207] In one implementation, scene-level hot words are obtained from the list of activated states in cache 2, and interface hot words are obtained from cache 3 according to the component name. Since scene-level hot words have been cached in cache 2 and interface hot words have been cached in cache 3 when the display interface is switched to the foreground, when receiving user voice, only system-level hot words, scene-level hot words and interface hot words need to be read from the cache, and there is no need to judge the scene in real time and parse the display interface, which improves the response speed during voice input.

[0208] After the hot word set is updated, the speech analysis module in the speech processing engine analyzes the user's speech and obtains the analysis text corresponding to the user's speech. The speech analysis module sends the analysis text to the speech recognition module. The speech recognition module performs semantic understanding and recognition on the analysis text and obtains the recognition content; that is, converts the analysis text into a natural language that is easier to understand. Exemplarily, the speech recognition module uses a natural language processing (NLP) algorithm to perform semantic analysis on the analysis text.

[0209] In one implementation, when performing semantic understanding and recognition, the speech recognition module compares the parsed text with the hot word set corresponding to the current display interface of the foreground application stored in cache 4. If the parsed text matches any hot word in the hot word set stored in cache 4, the speech recognition module successfully recognizes it and sends the recognition content to the instruction mapping module.

[0210] When performing semantic understanding and recognition, the parsed text is matched with the hot words. The two can have the same semantics or close semantics or the recognized content contains the hot words, etc. It is not necessary for the parsed text to be exactly the same as the hot words. For example, Figure 4B In the scenario shown, the user inputs the voice "return to desktop", and the system-level hot words include "return to desktop". "Return to desktop" and "return to desktop" have the same semantics, that is, they match. Since hot words are used in semantic understanding and recognition, semantic recognition has a reference standard, which improves the accuracy of speech recognition and semantic understanding.

[0211] In one implementation, the parsed text is first matched with the system-level hot words in the cache 4. If the match is successful (i.e., the parsed text matches any one of the system-level hot words in the cache 4), the recognition content is returned; if the parsed text fails to match the system-level hot words in the cache 4, the parsed text is matched with the scene-level hot words in the cache 4. If the parsed text matches any one of the scene-level hot words in the cache 4, the recognition content is returned; if the parsed text fails to match the scene-level hot words in the cache 4, the parsed text is matched with the interface hot words in the cache 4. If the parsed text matches any one of the interface hot words in the cache 4, the recognition content is returned; if the parsed text fails to match the interface hot words in the cache 4, the speech recognition fails.

[0212] That is to say, when performing speech recognition, the parsed text is matched with the system-level hot words first; if the match fails in the system-level hot words, it is matched with the scene-level hot words; if the match fails in the scene-level hot words, it is matched with the interface hot words. The priority of the system-level hot words is greater than the priority of the scene-level hot words, and the priority of the scene-level hot words is greater than the priority of the interface hot words. In this way, speech parsing and recognition can be closer to the user's true intention and avoid misoperation. For example, in the e-book scenario, the text in the display interface of the e-book application contains "come to the desktop". When the terminal device receives the user's voice input "return to the desktop", the parsed text "return to the desktop" is first matched with the system-level hot words. The system-level hot words include "return to the desktop", so that the terminal device can execute the corresponding instruction of "return to the desktop" and display the desktop interface. Instead of matching the parsed text "return to the desktop" with the text "come to the desktop" in the display interface of the e-book application.

[0213] For further information, please refer to Fig.11 The speech recognition module sends the recognition content to the instruction mapping module, which receives the recognition content and converts it into instructions that the terminal device can recognize and execute. The instruction mapping module sends the instruction to the multi-mode control module, which processes the instruction.

[0214] In one implementation, the instruction mapping module simulates and generates corresponding control events according to the identified content; and sends the control events as instructions to the multi-mode control module. The multi-mode control module sends the control events to the corresponding application for processing. Figure 7 , the mobile phone 100 displays the application interface 122 of the e-book application, and the current foreground application is the e-book application. The control "next page" is not set on the application interface 122. After receiving the user's voice "next page", the instruction mapping module generates a click event of the control "next page" according to the recognition content "next page". The instruction mapping module sends the click event of the control "next page" to the multi-mode control module, and the multi-mode control module sends the click event of the control "next page" to the e-book application for processing.

[0215] In one implementation, the instruction mapping module sends the instruction to the multi-mode control module, and the multi-mode control module calls the software development kit (SDK) of the relevant application to implement the logic of executing the instruction. Figure 7 , the mobile phone 100 displays the application interface 122 of the e-book application, and the current foreground application is the e-book application. After receiving the user voice "next page", the instruction mapping module generates a machine instruction to display the next page according to the recognition content "next page". The instruction mapping module sends the machine instruction to display the next page to the multi-mode control module. The multi-mode control module calls the SDK of the e-book application to implement the logic of switching the display interface of the e-book application to the next page of the current display interface.

[0216] The method for controlling a terminal device by voice provided in an embodiment of the present application is that when the terminal device receives the user's voice, the hot word set corresponding to the foreground application is obtained from the cache, and the hot word set includes system-level hot words, scene-level hot words corresponding to the foreground application, and interface hot words corresponding to the display interface of the foreground application. Hot words include not only texts in the display interface, but also system-level hot words and scene-level hot words. The terminal device supports more and richer hot words. Moreover, the hot words are obtained from the cache, and there is no need to parse them in real time from the current display interface. The rate of voice recognition is faster, and the terminal device responds to the user's voice faster, which improves the user's experience. In particular, when the display interface switches back and forth frequently, since the interface hot words corresponding to multiple display interfaces are stored in the cache, it is not necessary to frequently parse the text from the display interface again when the display interface switches, which reduces the power consumption generated by parsing the text and improves the response speed of the terminal device. If the display interface switches, and the scene corresponding to the application remains unchanged, there is no need to update the scene-level hot words, which improves the processing efficiency.

[0217] For example, Fig.13 A flow chart of a method for controlling a terminal device by voice provided in an embodiment of the present application is shown. Fig.13 As shown, the method includes:

[0218] The interface monitoring module of the terminal device monitors the application switching or the display interface switching within the application, and the interface monitoring module notifies the cache management module to update the cache. The cache management module obtains the list item corresponding to the previous display interface in cache 3 according to the component name of the previous display interface of the current display interface, and updates the cumulative usage time (totalUseTime) in the list item corresponding to the previous display interface in the cache (cache 3). The cache management module identifies the scene corresponding to the foreground application, and if the scene changes, marks the scene-level hot words corresponding to the foreground application in cache 2. If the scene has not changed, cache 2 does not need to be updated. The cache management module determines whether there is a list item corresponding to the current display interface in the cache (cache 3). If there is a list item corresponding to the current display interface in the cache (cache 3), update the last use time (lastUseTime) in the list item corresponding to the current display interface in the cache (cache 3); if there is no list item corresponding to the current display interface in the cache (cache 3), first determine whether the number of list items in the cache (cache 3) reaches the preset threshold. If the number of list items in the cache (cache 3) reaches the preset threshold, clear one or more list items in the cache (cache 3); for example, clear the N list items with the shortest cumulative use time according to the cumulative use time (totalUseTime) in the list items. If the number of list items in the cache (cache 3) does not reach the preset threshold, the cache management module obtains the text in the current display interface and generates interface hot words based on the obtained text; and creates a list item corresponding to the current display interface in the cache (cache 3), and saves the interface hot words in the list item. In this way, the cache update process is completed, the scene-level hot words corresponding to the current interface (current application) are marked in cache 2, and the interface hot words are saved in cache 3.

[0219] The user can input voice to the terminal device, and the application interaction module of the terminal device receives the user voice and determines whether the display interface of the terminal device switches from the last time the user voice was received to the current time the user voice was received. If the display interface of the terminal device switches, the voice processing engine obtains the hot word set (including system-level hot words, scene-level hot words and interface hot words) corresponding to the current display interface from cache 1, cache 2 and cache 3. Optionally, if cache 3 is empty, or there is no list item corresponding to the current display interface in cache 3, the application interaction module notifies the cache management module to update the cache. Further, the voice processing engine parses and recognizes the user voice. If the display interface of the terminal device does not switch, that is, the user voice received this time is not the first time the user voice is received on the current display interface, the voice processing engine directly parses and recognizes the user voice, and there is no need to obtain the hot word set again. If the recognition content of the user voice successfully matches any hot word in the hot word set, the instruction corresponding to the user voice is executed. Optionally, if the recognition content of the user voice fails to match the hot word set, the cache management module is notified to update the cache.

[0220] It is understandable that, in order to realize the above functions, the above terminal device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0221] The embodiment of the present application can divide the functional modules of the above-mentioned terminal device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0222] In the case of an integrated unit, Fig.14 A possible structural diagram of a terminal device involved in the above embodiment is shown. The terminal device 1400 includes: a processing unit 1401 , a storage unit 1402 , an audio unit 1403 and a display unit 1404 .

[0223] The processing unit 1401 is used to control and manage the actions of the terminal device 1400. For example, it can be used to update the scene-level hot words and interface hot words in the cache when an application switch or an interface switch within an application occurs; it can be used to parse and recognize the user voice when receiving the user voice, and / or used for other processes of the technology described in this article.

[0224] The storage unit 1402 is used to store program codes and data of the terminal device 1400. For example, it stores system-level hot words, scene-level hot words, interface hot words, etc. The processing unit 1401 calls the program code stored in the storage unit 1402 to execute each step in the above method embodiment.

[0225] The audio unit 1403 is used to collect the voice uttered by the user and play the voice.

[0226] The display unit 1404 is used to display the interface of the terminal device 1400 .

[0227] Of course, the unit modules in the terminal device 1400 include but are not limited to the processing unit 1401, the storage unit 1402, the audio unit 1403 and the display unit 1404. For example, the terminal device 1400 may also include a power supply unit, a communication unit, etc. The power supply unit is used to supply power to the terminal device 1400. The communication unit is used to support communication between the terminal device 1400 and other devices.

[0228] Among them, the processing unit 1401 can be a processor or a controller, for example, a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The processor may include an application processor and a baseband processor. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The storage unit 1402 may be a memory. The audio unit 1403 may include a microphone, a speaker, a receiver, etc. The display unit 1404 may be a display screen, etc. The communication unit may be a transceiver, a transceiver circuit or a communication interface, etc.

[0229] For example, the processing unit 1401 is a processor (such as Fig. 9 The processor 110 shown in FIG. 14 ), the storage unit 1402 may be a memory (eg, Fig. 9 The audio unit 1403 may include a microphone (eg, Fig. 9 Microphone 170C shown), speaker (such as Fig. 9 Speaker 170A shown), receiver (such as Fig. 9 The display unit 1404 may be a display screen (eg, Fig. 9 The communication unit includes a mobile communication module (such as Fig. 9 The mobile communication module 150 shown in FIG. Fig. 9 The mobile communication module and the wireless communication module may be collectively referred to as a communication interface. The terminal device 1400 provided in the embodiment of the present application may be Fig. 9The terminal device 100 shown in FIG. 1 is a schematic diagram of a terminal device 100. The processor, memory, microphone, display screen, and communication interface, etc., may be coupled together, for example, via a bus. The processor calls the program code stored in the memory to execute each step in the above method embodiment.

[0230] The present application also provides a chip system (eg, a system on a chip (SoC)), such as Fig.15 As shown, the chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 and the interface circuit 1502 can be interconnected via lines. For example, the interface circuit 1502 can be used to receive signals from other devices (such as a memory of a terminal device). For another example, the interface circuit 1502 can be used to send signals to other devices (such as a processor 1501 or a touch screen of a terminal device). Exemplarily, the interface circuit 1502 can read instructions stored in the memory and send the instructions to the processor 1501. When the instructions are executed by the processor 1501, the terminal device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0231] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned terminal device, the terminal device executes each function or step in the above-mentioned method embodiment.

[0232] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0233] Among them, the terminal device 1400, chip system, computer-readable storage medium or computer program product provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0234] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0235] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0236] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0237] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0238] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0239] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for controlling a terminal device by voice, applied to a terminal device, wherein the terminal device includes a microphone, and is characterized in that: The method comprises: The terminal device displays a first interface, wherein the first interface includes a first switch; In response to the user turning on the first switch, the terminal device turns on the voice control function; the turning on the voice control function includes: turning on the recording channel corresponding to the voice control function, the recording channel being used to obtain the user voice collected by the microphone; Receiving a first voice input by a user when the terminal device displays a second interface, and the terminal device executes a first instruction corresponding to the first voice; wherein the second interface does not include a first text corresponding to the first instruction; In response to the user's operation of turning off the first switch, the terminal device turns off the voice control function; turning off the voice control function includes: turning off the recording channel corresponding to the voice control function.

2. The method according to claim 1, characterized in that The terminal device starts the voice control function including: The terminal device displays a first recording icon and a second recording icon, the first recording icon indicates that the voice control function is turned on, and the second recording icon indicates that the recording channel is turned on.

3. The method according to claim 1 or 2, characterized in that: The first voice matches at least one preset hot word.

4. The method according to claim 3, characterized in that The hot words include at least one of the following: Swipe left, swipe right, swipe up, return to the desktop, return.

5. The method according to claim 3 or 4, characterized in that: The application to which the second interface belongs belongs to the audio category, the short video category, or the long video category, and the hot words include at least one of the following: Play, pause, stop, fast forward, rewind.

6. The method according to claim 3 or 4, characterized in that: The application to which the second interface belongs belongs to the category of electronic book playback, and the hot words include at least one of the following: Previous page, next page, table of contents, next chapter.

7. The method according to any one of claims 3 to 6, characterized in that: Before receiving the first voice input by the user when the terminal device displays the second interface, the method further includes: When it is detected that the second interface is switched to the foreground display, the terminal device updates the hot words corresponding to the second interface in the cache; After receiving the first voice input by the user when the terminal device displays the second interface, the method further includes: The terminal device obtains the hot word corresponding to the second interface from the cache.

8. The method according to claim 7, characterized in that The terminal device updates the hot words corresponding to the second interface in the cache, including: The terminal device obtains a scene-level hot word corresponding to the second interface; the scene-level hot word is applicable to all applications in a scene; wherein the scene is pre-configured according to the type of application, and the correspondence between the scene and the scene-level hot word is pre-configured; The terminal device marks the scene-level hot words corresponding to the second interface in the cache.

9. The method according to claim 8, characterized in that The cache of the terminal device includes at least one list item, each list item is used to store an interface hot word corresponding to an interface, and the interface hot word is applicable to an interface. The terminal device updates the hot words corresponding to the second interface in the cache, including: If the list item corresponding to the second interface does not exist in the cache of the terminal device, the terminal device creates a list item corresponding to the second interface in the cache; the list item corresponding to the second interface is used to store the interface hot words corresponding to the second interface.

10. The method according to claim 9, characterized in that After the terminal device creates a list item corresponding to the second interface in the cache, the method further includes: The terminal device acquires text included in the second interface; The terminal device saves the interface hot words corresponding to the second interface in the list item corresponding to the second interface according to the text included in the second interface.

11. The method according to claim 10, characterized in that Before the terminal device creates a list item corresponding to the second interface in the cache, the method further includes: If the number of list items in the cache is greater than or equal to a preset threshold, one or more list items in the cache are cleared.

12. The method according to claim 11, characterized in that The clearing one or more list items in the cache includes: Clear the list items corresponding to one or more interfaces in the cache that have the shortest total cumulative calling time.

13. The method according to any one of claims 7 to 12, characterized in that: The first voice is the user voice received for the first time after the second interface switches to the foreground display.

14. The method according to claim 9, characterized in that The terminal device obtains the hot word corresponding to the second interface from the cache, including: The terminal device obtains system-level hot words, scene-level hot words and interface hot words corresponding to the second interface from the cache; the system-level hot words are pre-configured and applicable to any application on the terminal device.

15. The method according to claim 14, characterized in that The method further comprises: The terminal device obtains the parsed text of the first speech; The terminal device performs voice recognition on the parsed text of the first voice according to the hot words corresponding to the second interface.

16. The method according to claim 15, characterized in that The terminal device performs speech recognition on the parsed text of the first speech according to the hot words corresponding to the second interface, including: The terminal device matches the parsed text of the first voice with the system-level hotword; If the parsed text of the first voice fails to match the system-level hotword, matching the parsed text of the first voice with the scene-level hotword corresponding to the second interface; If the parsed text of the first voice fails to match the scene-level hot words corresponding to the second interface, the parsed text of the first voice is matched with the interface hot words corresponding to the second interface.

17. A terminal device, characterized in that: The terminal device includes: a processor, a memory, a microphone and a display screen, the microphone is used to collect user voice, the display screen is used to display the interface of the terminal device, the processor is coupled to the memory; the memory is used to store computer program code; the computer program code includes computer instructions, and when the processor executes the above-mentioned computer instructions, the terminal device executes the method described in any one of claims 1-16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 16.

Images