Method for processing concurrency conflict of input voice and terminal equipment

Abstract

The invention discloses an input voice concurrency conflict processing method and terminal equipment, and relates to the technical field of terminals. After the first voice control function is started, in response to user operation, the terminal equipment is awakened, and the second voice control function is started. After the second voice control function is started, the first voice control function is temporarily unavailable. If the terminal equipment executes one voice instruction through the second voice control function, or the terminal equipment does not receive an instruction issued by the user through voice within a period of time after being awakened, the second voice control function is closed. After the second voice control function is closed, the first voice control function is automatically started. That is to say, when other applications collect the user voice through the microphone, the first voice control function is suspended; therefore, the first voice control can be prevented from being triggered by mistake to influence normal use of the terminal equipment by a user.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method and terminal device for processing concurrent conflicts in recorded voice. Background Art

[0002] With the development of terminal technology, terminal devices are increasingly used in people's daily lives. In order to improve the operating efficiency of users using terminal devices, terminal devices generally provide a variety of human-computer interaction methods such as human-computer interaction interface, voice control, gesture control, etc. Among them, voice control is a major and quick human-computer interaction method. In particular, when users are driving, cooking, reading, and other scenarios where their hands are occupied, it is very convenient and quick to control the terminal device by voice.

[0003] When the voice control function is turned on on a terminal device, the user's voice will be collected through the microphone. There are also some applications or functions on the terminal device, such as recording applications, phone applications, audio and video call applications, etc., which also record voice through the microphone. If applications or functions that need to record voice collect voice at the same time, concurrency conflicts may occur. How to handle the concurrency conflicts of multiple applications or functions recording voice is a problem that needs to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method and a terminal device for processing concurrent conflicts in recorded voice, which can resolve concurrent conflicts in recorded voice for multiple applications or functions, avoid abnormal processing by the terminal device, and improve the user experience of using the voice control function.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for handling concurrent conflicts in recorded voice is provided, the method comprising: a terminal device displays a first switch of a first interface; in response to a user turning on the first switch, the terminal device displays a first prompt icon, indicating that a first voice control function has been turned on; after the first voice control function is turned on, the user can control the terminal device by voice; for example, after receiving a first voice collected by a microphone, the terminal device executes a first instruction corresponding to the first voice. The first voice does not include a wake-up word of the terminal device; that is, the first voice control function is not a voice assistant, for example, the first voice control function is visible and can be spoken.

[0007] When the first voice control function is turned on, in response to the user's operation of turning on the second voice control function, the terminal device stops displaying the first prompt icon, displays the second prompt icon, and displays the first prompt card; the second prompt icon indicates that the first voice control function is temporarily unavailable, and the first prompt card indicates that the second voice control function is turned on. In other words, in response to the second voice control function being turned on, the first voice control function is temporarily unavailable.

[0008] When the second voice control function is turned on, the user can control the terminal device through voice; for example, after receiving the second voice collected by the microphone, the terminal device executes the second instruction corresponding to the second voice; after the terminal device executes the second instruction corresponding to the second voice, the second voice control function is turned off and the first prompt card stops being displayed.

[0009] After executing the second instruction corresponding to the second voice, the second voice control function is turned off and the first voice control function is automatically turned on; the terminal device stops displaying the second prompt icon and displays the first prompt icon, indicating that the first voice control function is turned on.

[0010] In this method, after the visible and speak function (first voice control function) has been turned on, in response to the user operation, the terminal device is awakened and the voice assistant function (second voice control function) is turned on. The voice assistant function occupies the microphone, and the visible and speak function is temporarily unavailable. After the voice assistant function is turned on, the user can issue a command to the terminal device by inputting voice to the terminal device, and the terminal device executes the command corresponding to the voice. After the terminal device executes a command, or the terminal device does not receive a command issued by the user through voice within a period of time (for example, within 8 seconds) after being awakened, the voice assistant function is turned off. The voice assistant function no longer occupies the microphone, and the visible and speak function is automatically turned on.

[0011] After the first voice control function is turned on, the user's voice will be continuously collected through the microphone; if the user's voice is mistakenly distributed to the first voice control function when other applications collect the user's voice through the microphone, the first voice control function will be mistakenly triggered to execute the command corresponding to the voice. In this method, when other applications (for example, the second voice control function) collect the user's voice through the microphone, the visible and speak function (the first voice control function) is suspended (the microphone is exited from occupation). This can avoid the first voice control function being mistakenly triggered to execute voice commands and affecting the user's normal use of the terminal device.

[0012] In combination with the first aspect, in a possible implementation, in response to the user turning on the first switch, the terminal device turns on a recording channel corresponding to the first voice control function, and the recording channel is used to obtain the user voice collected by the microphone.

[0013] That is, in response to the user turning on the first switch, the terminal device turns on the first voice control function. The terminal device turns on the recording channel corresponding to the first voice control function, that is, turns on the first voice control function, so that the audio stream collected by the microphone will be distributed to the first voice control function. After turning on the first voice control function, the terminal device displays a first prompt icon to prompt the user that the first voice control function has been turned on.

[0014] In combination with the first aspect, in a possible implementation, in response to the user's operation of turning on the second voice control function, the terminal device turns on the recording channel corresponding to the second voice control function and turns off the recording channel corresponding to the first voice control function.

[0015] In this method, the second voice control function is turned on and the first voice control function is turned off by switching the recording channel. After the recording channel corresponding to the second voice control function is turned on and the recording channel corresponding to the first voice control function is turned off, the audio stream collected by the microphone will be distributed to the second voice control function instead of the first voice control function.

[0016] In combination with the first aspect, in a possible implementation, after executing the second instruction corresponding to the second voice, the terminal device closes the recording channel corresponding to the second voice control function and opens the recording channel corresponding to the first voice control function.

[0017] In this method, after the terminal device is awakened and a voice command is executed, the second voice control function is automatically turned off. After the second voice control function is turned off, the first voice control function is automatically turned on. The audio stream collected by the microphone is distributed to the first voice control function, but not to the second voice control function.

[0018] In combination with the first aspect, in a possible implementation, after the terminal device stops displaying the second prompt icon and displays the first prompt icon, in response to the user's operation of turning on the first function, the terminal device turns on the first function; in response to the first function being turned on, the terminal device stops displaying the first prompt icon and displays the second prompt icon. After the first function is turned on, the terminal device collects the user's voice through a microphone. In response to the user's operation of turning off the first function, the terminal device stops displaying the second prompt icon and displays the first prompt icon.

[0019] Among them, the terminal device turning on the first function includes: the terminal device turning on the recording channel corresponding to the first function.

[0020] In combination with the first aspect, in a possible implementation, in response to the first function being turned on, the terminal device turns off the recording channel corresponding to the first voice control function.

[0021] In this method, when other applications (for example, the first function) collect user voice through the microphone, the visible and speak function (the first voice control function) is suspended (exit occupying the microphone), which can avoid false triggering of voice control and affecting the user's normal use of the terminal device.

[0022] In combination with the first aspect, in a possible implementation, in response to the user turning off the first switch, the terminal device turns off the recording channel corresponding to the first voice control function.

[0023] In combination with the first aspect, in a possible implementation, the operation of enabling the second voice control function includes: long pressing the power button on the terminal device, or voice inputting a wake-up word.

[0024] In combination with the first aspect, in a possible implementation, when the terminal device displays the first prompt icon, the terminal device receives a third voice collected by a microphone; the third voice includes a wake-up word of the terminal device; the terminal device continues to display the first prompt icon. And the terminal device obtains the user voice through the recording channel corresponding to the first voice control function.

[0025] In this method, the terminal device displays a first prompt icon indicating that the first voice control function has been turned on. The terminal device receives the wake-up word input by the user's voice, and the first voice control function is still available. The display interface of the terminal device does not change, and it is still the prompt icon of the first voice control function, and the recording channel is not switched.

[0026] In a second aspect, a method for handling concurrent conflicts in recorded voice is provided, the method comprising: a terminal device displays a first interface, the first interface comprising a first switch; in response to a user operation of turning on the first switch, the terminal device turns on a recording channel corresponding to a voice control function; the recording channel is used to obtain the user voice captured by the microphone; after receiving the first voice captured by the microphone, the terminal device executes a first instruction corresponding to the first voice; in response to the user's first operation, the terminal device establishes a recording channel corresponding to a first application, and closes the recording channel corresponding to the voice control function.

[0027] In this method, when the first voice control function has started, if the first application occupies the microphone and collects the user's voice through the microphone, the first voice control function (visible and speak function) is suspended (exits occupying the microphone), which can avoid false triggering of voice control and affecting the user's normal use of the terminal device.

[0028] In combination with the second aspect, in a possible implementation, in response to the user turning on the first switch, the terminal device displays a first prompt icon, and the first prompt icon indicates that the voice control function has been turned on.

[0029] In combination with the second aspect, in a possible implementation, in response to a first operation of the user, the terminal device stops displaying the first prompt icon and displays a second prompt icon, where the second prompt icon indicates that the voice control function is temporarily unavailable.

[0030] In combination with the second aspect, in a possible implementation, in response to the user's second operation, the terminal device closes the recording channel corresponding to the first application; in response to the recording channel corresponding to the first application being closed, the terminal device opens the recording channel corresponding to the voice control function.

[0031] According to a third aspect, a method for handling concurrent conflicts in recorded voice is provided, the method comprising: in response to a first operation of a user, a terminal device establishes a recording channel corresponding to a first application; the recording channel is used to obtain user voice captured by a microphone; the terminal device displays a first interface, the first interface includes a first switch; the first switch is a switch corresponding to a voice control function; an operation of turning on the first switch by a user is received; in response to the operation of turning on the first switch by the user, the terminal device displays a prompt icon, indicating that the voice control function is temporarily unavailable.

[0032] In this method, when receiving an operation of turning on the first switch by the user to enable the voice control function (the see-and-speak function), if other applications (such as recording applications, phone applications, audio and video call applications, etc.) are collecting user voice through a microphone, the terminal device temporarily does not enable the see-and-speak function, and the see-and-speak function is temporarily unavailable. This can avoid erroneous triggering of voice control and affect the user's normal use of the terminal device.

[0033] In combination with the third aspect, in a possible implementation, in response to the user's second operation, the terminal device closes the recording channel corresponding to the first application; in response to the recording channel corresponding to the first application being closed, the terminal device opens the recording channel corresponding to the voice control function.

[0034] In this method, when the first application exits and occupies the microphone, the voice control function (see-and-speak function) is automatically turned on.

[0035] In combination with the third aspect, in a possible implementation, after the terminal device turns on the recording channel corresponding to the voice control function, when receiving the first voice collected by the microphone, the terminal device executes the first instruction corresponding to the first voice.

[0036] In a fourth aspect, a terminal device is provided, comprising: a processor and a memory; the memory is used to store computer execution instructions, and when the terminal device is running, the processor executes the computer execution instructions stored in the memory to enable the terminal device to perform a method as described in any one of the first aspect, the second aspect or the third aspect above.

[0037] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer can execute any of the methods in the first aspect, the second aspect or the third aspect.

[0038] Among them, the technical effects brought about by any design method in the fourth and fifth aspects can refer to the technical effects brought about by different design methods in the first, second or third aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1A A schematic diagram of a scenario example of a method for controlling a terminal device by voice;

[0040] Figure 1B A schematic diagram of another scenario example of a method for controlling a terminal device by voice;

[0041] Figure 2 A schematic diagram of a scenario example of setting a preset switch for a voice control function provided in an embodiment of the present application;

[0042] Figure 3A Schematic diagram 1 of an example of a voice control scenario applicable to the method for handling concurrent conflicts of recorded voices provided in an embodiment of the present application;

[0043] Figure 3B A voice control scenario example diagram applicable to the method for handling concurrent conflicts of recorded voices provided in the embodiment of the present application Figure 2 ;

[0044] Figure 3C Schematic diagram 3 of a voice control scenario example applicable to the method for handling concurrent conflicts of recorded voices provided in an embodiment of the present application;

[0045] Figure 4A A schematic diagram of the software architecture of a terminal device provided in an embodiment of the present application;

[0046] Figure 4B This is a schematic diagram of the interaction flow of each module during the activation and use of the voice assistant function;

[0047] Figure 4C A schematic diagram of the interaction flow of each module during the activation and use of the Visible is Talkable function;

[0048] Figure 4D This is a schematic diagram of the interaction flow of each module during the activation and use of the recording function;

[0049] Figure 5 A schematic diagram of a scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0050] Figure 6 A scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application Figure 2 ;

[0051] Figure 7 Schematic diagram 3 of a scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0052] Figure 8 Schematic diagram 4 of a scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0053] Fig. 9 A scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application Figure 5 ;

[0054] Fig.10 A scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application Figure 6 ;

[0055] Fig.11 A flowchart of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0056] Fig.12 A flow chart of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application Figure 2 ;

[0057] Fig.13 A flowchart diagram of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application is shown in FIG3;

[0058] Fig.14 A fourth flowchart of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0059] Fig.15 This is a schematic diagram of the interaction flow of each module when the voice assistant function and the see-and-speak function are used concurrently;

[0060] Fig.16 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0061] Fig.17 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0062] Fig.18 A schematic diagram of the chip system structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0064] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include such expressions as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0065] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0066] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0067] Voice control function:

[0068] Many terminal devices support voice control. The terminal device collects user voice through a microphone, analyzes and recognizes the user voice, and executes the instructions corresponding to the user voice, so that the user can control the terminal device through voice.

[0069] Generally speaking, in order to save power consumption of terminal devices and avoid false triggering, the voice control function needs to be turned on before it can be used. For example, the user inputs a preset word (called a wake-up word) to the terminal device through voice to wake up the terminal device. After the terminal device is awakened, it can execute the instructions corresponding to the user's voice, that is, the voice control function is turned on. For example, the user can turn on or off the voice control function by turning on or off the preset switch in the human-computer interaction interface of the terminal device.

[0070] In different terminal devices, the voice control function may have different names, such as "voice control", "intelligent voice", "voice assistant", "see and speak", "voice command", "free command", "intelligent AI", etc. The specific implementation of voice control functions with different names may also be different.

[0071] The following is an exemplary introduction to several different implementations of the voice control function.

[0072] Voice Assistant:

[0073] Before the user uses the voice assistant to control the terminal device, the terminal device needs to be woken up first. In one example, the wake-up word input by the user to the terminal device is detected, and the terminal device is awakened. In another example, the terminal device is awakened by detecting the user's long press of the power button. In another example, the terminal device is awakened by detecting the breath generated when the user inputs voice to the terminal device. Generally speaking, before the terminal device is awakened, the microphone of the terminal device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone is only detected for wake-up words at the kernel layer, and the corresponding recording channel of the voice assistant is not started in the system and driver of the terminal device.

[0074] In response to user operations (such as receiving a wake-up word input by the user's voice), the terminal device is awakened, and the corresponding recording channel of the voice assistant is started in the system and driver. After the terminal device is awakened, the voice (audio stream) collected by the microphone is sent to the voice assistant application for processing through the recording channel corresponding to the voice assistant. In this way, the terminal device can execute the instructions corresponding to the user's voice, enabling the user to control the terminal device through voice; it can also realize functions such as dialogue with the user.

[0075] For example, Figure 1A The figure shows a schematic diagram of a scenario in which a user uses a voice assistant to control a mobile phone. Figure 1AAs shown, the mobile phone displays the desktop interface, and the user inputs the voice "Hello YOYO" into the mobile phone. In response to receiving the wake-up word "Hello YOYO", the mobile phone is awakened. Exemplarily, after the mobile phone is awakened, the voice "I am here" is played to remind the user that the mobile phone has been awakened. In one example, after the mobile phone is awakened, a voice assistant prompt card is displayed, indicating that the mobile phone has been awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. Exemplarily, as Figure 1A As shown, the user inputs the voice "open video" to the mobile phone, and the mobile phone analyzes and recognizes the user's voice and executes the command corresponding to the voice "open video". Exemplarily, the mobile phone starts the video application in response to receiving the user's voice "open video".

[0076] For example, Figure 1B FIG. 1 shows another example of a scenario in which a user uses a voice assistant to control a mobile phone. Figure 1B As shown, the mobile phone displays the desktop interface, and the user long presses the power button. In response to receiving the user's long press of the power button operation, the mobile phone is awakened. In one example, after the mobile phone is awakened, a voice assistant prompt card is displayed, indicating that the mobile phone has been awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. For example, Figure 1B As shown, the user inputs the voice "open video" to the mobile phone, and the mobile phone analyzes and recognizes the user's voice and executes the command corresponding to the voice "open video". Exemplarily, the mobile phone starts the video application in response to receiving the user's voice "open video".

[0077] In some implementations, after the terminal device is awakened, the user can issue a command to the terminal device by inputting voice into the terminal device, and the terminal device executes the command corresponding to the voice. After the terminal device executes a command, or if the terminal device does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after being awakened, the terminal device no longer responds to the command issued by the user through voice. For example, the terminal device will close the recording channel corresponding to the voice assistant. The user needs to input the wake-up word to the terminal device again to wake up the terminal device before he can issue a command to the terminal device by inputting voice into the terminal device again. In other words, after the terminal device is awakened, it enters a "short reception" state and can respond to the user's voice command within a short period of time (for example, within 8 seconds).

[0078] In some implementations, when the terminal device is connected to the network, it supports entering a continuous conversation scenario after being awakened, and the user can have a continuous conversation with the terminal device. After each broadcast of the terminal device is completed, it will continue to pick up the sound without repeated awakening. Until the user exits the continuous conversation through instructions such as "exit".

[0079] It can be seen that:

[0080] It can be said that it is implemented locally by the terminal device and does not need to be connected to the network.

[0081] Visible and Talkable is controlled by a preset switch. Users can turn on Visible and Talkable by turning on the preset switch, and can exit Visible and Talkable by turning off the preset switch.

[0082] Taking the terminal device as a mobile phone as an example, for example, Figure 2 As shown, the user can open the settings function of the mobile phone 100; for example, the user clicks on the application icon of the "Settings" application on the desktop. In response to the user clicking on the application icon of the "Settings" application, the mobile phone 100 displays the "Settings" interface 101. The "Settings" interface 101 includes a "Smart Voice" option 102, and the "Smart Voice" option 102 is used to set the smart voice function. Exemplarily, in response to the user clicking on the "Smart Voice" option 102, the mobile phone 100 displays the "Smart Voice" interface 103, and the "Smart Voice" interface 103 includes a "Visible and Speakable" option 104. The user can click on the "Visible and Speakable" option 104 to set options related to the Visible and Speakable function. Exemplarily, refer to Figure 2 In response to the user's click operation on the "See and Say" option 104, the mobile phone 100 displays the "See and Say" interface 105. Optionally, the "See and Say" interface 105 includes a prompt message 106 for prompting the user how to use the "See and Say" function. The "See and Say" interface 105 also includes a "See and Say" switch 107. The user can click the "See and Say" switch 107 to turn on or off the "See and Say" switch. In one example, in response to receiving the user's click operation on the "See and Say" switch 107, the "See and Say" switch of the mobile phone 100 is turned on to turn on the "See and Say" function. Optionally, the "See and Say" interface 105 displays a prompt message 108 for prompting the user that the "See and Say" function has been successfully turned on.

[0083] In one implementation, after the see-and-speak function is turned on, the mobile phone 100 displays a first prompt icon, which indicates that the see-and-speak function is turned on. Figure 2 As shown, after the "see and speak" switch 107 is turned on, the status bar of the display interface of the mobile phone 100 displays a prompt icon 10a, indicating that the see and speak function has been turned on.

[0084] In one scenario, the preset switch corresponding to Visible and Talk on the terminal device is not turned on, and the microphone of the terminal device is not turned on. When the preset switch corresponding to Visible and Talk is turned on, the terminal device starts the microphone and starts the corresponding recording channel of Visible and Talk in the system and driver. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talk application for processing through the corresponding recording channel of Visible and Talk, so that the user can control the terminal device through voice.

[0085] In another scenario, the preset switch corresponding to Visible and Talking of the terminal device is not turned on, and the microphone of the terminal device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. When the preset switch corresponding to Visible and Talking is turned on, the corresponding recording channel of Visible and Talking is started in the system and driver of the terminal device. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talking application for processing through the corresponding recording channel of Visible and Talking, so that the user can control the terminal device through voice.

[0086] After Visible is Talking is turned on, the corresponding recording channel of Visible is Talking is started in the system and driver of the terminal device, and the terminal device enters a "long receiving" state to continuously collect surrounding sounds. Users can give commands to the terminal device at any time through voice without having to enter a wake-up word to wake up the terminal device.

[0087] In one implementation, after any function on the terminal device turns on the voice recording function of the terminal device (turns on the microphone and establishes a recording channel), the terminal device will send a prompt message to the user to remind the user that the terminal device is in a voice collection state. In this way, the user's privacy can be prevented from being leaked. For example, after the visible and audible function is turned on, the terminal device enters a state of continuous voice collection, and a third prompt icon is displayed on the display interface of the terminal device. The third prompt icon indicates that the recording channel is turned on, which is used to remind the user that the microphone is collecting voice. For example, Figure 2 As shown, the status bar of the display interface of the mobile phone 100 displays a prompt icon 10b, indicating that the recording channel is turned on, which is used to prompt the user that the microphone is collecting voice.

[0088] After the See and Speak function is turned on, the terminal device establishes a recording channel and continuously collects the user's voice through the microphone. The user can input voice into the terminal device at any time. The terminal device analyzes and recognizes the user's input voice and executes the instructions corresponding to the user's voice.

[0089] In one example, if Figure 3AAs shown, the mobile phone 100 displays a desktop interface 120. The user inputs a voice message "open video" to the mobile phone 100. In response to receiving the voice message "open video" input by the user, the mobile phone 100 starts the video application and displays a user interface 121 of the video application.

[0090] In another example, Figure 3B As shown, the mobile phone 100 displays the desktop interface 120 and receives the user's voice input "swipe left". In response to receiving the user's voice "swipe left", the mobile phone 100 displays the negative one screen interface 122.

[0091] In another example, Figure 3C As shown, the mobile phone 100 displays the interface 123 of the short video application. The user inputs a voice "return to the desktop" to the mobile phone 100. In response to receiving the voice "return to the desktop" input by the user, the mobile phone 100 executes the instruction corresponding to the voice "return to the desktop" and displays the desktop interface 120.

[0092] It should be noted that the "collecting user voice through a microphone" and "occupying the microphone" described in the embodiments of the present application refer to the process of starting a recording channel in the system and driver of the terminal device and processing the audio stream of the user voice collected by the microphone. In other scenarios, the terminal device turns on the microphone to pick up the surrounding environment, but does not establish a recording channel in the system and driver, which does not belong to the cases of "collecting user voice through a microphone" and "occupying the microphone" described in the embodiments of the present application. For example, before the terminal device is awakened, the microphone of the terminal device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds in the surrounding environment. The voice collected by the microphone is only detected for wake-up words at the kernel layer, and the corresponding recording channel of the voice assistant is not started in the system and driver of the terminal device. This situation does not belong to the cases of "collecting user voice through a microphone" and "occupying the microphone" described in the embodiments of the present application.

[0093] The following is an illustrative introduction to the process of collecting user voice through a microphone for various applications and functions in the terminal device in combination with the software architecture of the terminal device.

[0094] The method for handling concurrent conflicts in recorded voice provided in the embodiment of the present application can be applied to a terminal device 100 including a microphone. The terminal device 100 may include a mobile phone, a tablet computer, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (e.g., a smart TV, a smart screen, a large screen, a smart speaker, a smart air conditioner, etc.), a personal digital assistant (PDA), a wearable device (e.g., a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a virtual reality device, etc., and the embodiment of the present application does not impose any restrictions on this.

[0095] In the embodiment of the present application, the terminal device 100 is a terminal device that can run an operating system and install application programs. Optionally, the operating system running on the terminal device can be system, system, System, etc.

[0096] In some embodiments, the software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture or a cloud architecture. Taking the system as an example, the software structure of the terminal device 100 is exemplarily described.

[0097] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0098] like Figure 4A As shown, the application layer may include a series of application packages APK. For example, applications such as camera, recording, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video call, short message, etc. The application layer also includes a voice processing engine for parsing, recognizing and processing voice. In an embodiment of the present application, the application layer includes a voice control APK, which is used to provide a voice control function (for example, a visible and speakable function and a voice assistant function) of the terminal device 100. Exemplarily, the voice control APK includes a visible and speakable APK and a voice assistant application; the visible and speakable APK is used to provide the visible and speakable function of the terminal device 100, and the voice assistant application is used to provide the voice assistant function of the terminal device 100; wherein, the visible and speakable APK includes a service presentation module, a service processing module, a voice processing module, etc.

[0099] Among them, the business presentation module is used to provide information displayed to the user on the user interface. For example, it provides prompt information displayed on the display interface of the terminal device 100. The business processing module is used to control the relevant process of recording voice when you can see and speak. For example, it controls the opening and closing of the visible and speaking function, controls the availability or unavailability of the visible and speaking function, etc. The business processing module may include a recording management module, a status management module, a monitoring module, etc. The monitoring module is used to monitor whether other applications occupy the microphone. The recording management module is used to control the opening and closing of the recording channel corresponding to the visible and speaking function; for example, according to the visible and speaking switch being turned on or off, the recording channel corresponding to the visible and speaking function is turned on or off. The status management module is used to manage the status of the visible and speaking function, such as turned on, turned off, temporarily unavailable, etc.

[0100] The voice processing module is used to recognize and process the collected user voice. For example, the voice processing module calls the voice processing engine to perform text parsing, semantic understanding and recognition on the audio stream collected by the microphone, and converts the user voice into voice commands. The voice processing module sends the voice commands to the multi-mode control module of the application framework layer, and the multi-mode control module distributes the voice commands to the corresponding application for execution.

[0101] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0102] like Figure 4A As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, an activity manager service (AMS), an application manager (PMS), a multi-mode control module, an audio module, and the like.

[0103] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0104] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0105] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0106] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0107] AMS is mainly responsible for the startup, switching, and scheduling of the four major components in the system, as well as the management and scheduling of application processes. Its responsibilities are similar to the process management and scheduling module in the operating system. When a process or component is started, the request will be passed to AMS through the Binder communication mechanism, and AMS will then handle it uniformly.

[0108] PMS handles package management related tasks, such as application installation, application uninstallation, etc. It is also used to provide all the information of an application, such as startup, shutdown, etc.

[0109] The multi-mode control module is used to manage the execution of voice commands, for example, sending voice commands to applications so that the applications execute the voice commands.

[0110] The audio module is used to handle audio-related functions, such as recording audio, playing audio, etc. It includes a recording module, which is used to manage the related process of audio recording.

[0111] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0112] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0113] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0114] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0115] A 2D graphics engine is a drawing engine for 2D drawings.

[0116] Android runtime is responsible for the scheduling and management of the Android system. Android runtime includes core libraries and virtual machines.

[0117] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0118] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0119] The kernel layer is the layer between hardware and software. The kernel layer can include display drivers, sensor drivers, audio drivers, Wi-Fi drivers, etc. The kernel layer also includes a wake-up word detection module, which is used to detect whether the wake-up word is received based on the audio collected by the microphone. The kernel layer also includes an audio digital signal processor (ADSP), which is used to distribute and process the audio collected by the microphone, such as establishing a recording channel for the application and distributing audio streams to the application.

[0120] Combination Figure 4A The software architecture shown, Figure 4B The interaction process of each module during the activation and use of the voice assistant function is shown.

[0121] Exemplary, reference Figure 4B After the voice assistant switch is turned on, the terminal device will turn on the microphone, and the microphone will work in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone will be transmitted to the wake-up word detection module for wake-up word detection. If the user inputs a wake-up word to the terminal device, the wake-up word detection module detects the wake-up word and notifies the voice assistant application that the terminal device is awakened. The voice assistant application starts the voice assistant function and requests the audio module to establish a recording channel corresponding to the voice assistant function. The audio module requests the ADSP to establish a recording channel. In this way, a recording channel corresponding to the voice assistant function is established. The audio stream collected by the microphone will be distributed to the voice assistant application for processing through the recording channel corresponding to the voice assistant function.

[0122] Optionally, after a preset time (e.g., 8 seconds), the recording channel corresponding to the voice assistant function is turned off.

[0123] Combination Figure 4A The software architecture shown, Figure 4C The interactive process of each module during the activation and use of the See-is-Say function is shown.

[0124] Exemplary, reference Figure 4C , the terminal device receives the user's operation of turning on the Visible and Talk switch, and turns on the Visible and Talk function. The Visible and Talk application requests the audio module to establish a recording channel corresponding to the Visible and Talk function. The audio module requests the ADSP to establish a recording channel. In this way, a recording channel corresponding to the Visible and Talk function is established. The audio stream collected by the microphone will be distributed to the Visible and Talk application for processing through the recording channel corresponding to the Visible and Talk function.

[0125] Similarly, other applications or functions on the terminal device can also request to establish a corresponding recording channel. Figure 4D , the terminal device receives the user's operation to start recording and starts the recording function. The recording application requests the audio module to establish a recording channel corresponding to the recording function. The audio module requests the ADSP to establish a recording channel. In this way, a recording channel corresponding to the recording function is established. The audio stream collected by the microphone will be distributed to the recording application for processing through the recording channel corresponding to the recording function.

[0126] After the voice control function is turned on, the terminal device establishes a recording channel in the system and driver, and the processing module for the voice control function obtains the audio stream collected by the microphone. In order to obtain the voice input by the user, parse and recognize the user's voice, execute the instructions corresponding to the user's voice, and enable the user to control the terminal device through voice. For example, after the Visible and Talking function is turned on, the terminal device starts the corresponding recording channel of Visible and Talking, and the microphone continues to collect the user's voice. The user can control the terminal device by voice. For example, after the voice assistant is turned on (the terminal device is awakened), the terminal device starts the corresponding recording channel of the voice assistant. Within a preset period of time (for example, within 8 seconds), the user can give a command to the terminal device by inputting voice into the terminal device, and the terminal device executes the instruction corresponding to the voice.

[0127] There are also some applications or functions on the terminal device, such as recording applications, telephone applications, audio and video call applications, etc., which will also establish recording channels and collect user voice through microphones.

[0128] If the "visible and speak" function is turned on, and the user uses the recording application, phone application, audio and video call application, voice assistant and other applications (or functions) with voice recording function on the terminal device, a concurrent conflict may occur, causing abnormal process of the terminal device. For example, the user uses the phone application to make a call, and the terminal device collects the user's voice. If the voice control function responds to the user's voice and executes the corresponding command, it will cause an erroneous operation. Exemplarily, when the "visible and speak" function is turned on, the user uses the mobile phone 100 to answer the call, and the microphone of the mobile phone 100 collects the user's voice. During the call, the user said "the cat climbed back to the desktop", and the voice control function of the mobile phone 100 mistakenly executed the command corresponding to the voice "return to the desktop", and the phone application was retreated to the background. Exemplarily, when the "visible and speak" function is turned on, the user uses the mobile phone 100 to make a call, and the microphone of the mobile phone 100 collects the user's voice. During the call, the user said "the strong wind broke the branches". The voice control function of the mobile phone 100 mistakenly executed the command corresponding to the voice "hang up", and the current call was hung up.

[0129] An embodiment of the present application provides a method for handling concurrent conflicts in voice recording. When other applications collect user voice through a microphone, the visible and speak function is suspended (exit occupying the microphone) to avoid false triggering of voice control and affecting the user's normal use of the terminal device.

[0130] Establishing a recording channel In some embodiments, when the Visible and Talking function is turned on and the Visible and Talking function collects the user's voice through the microphone, other applications (e.g., recording applications, phone applications, audio and video call applications, etc.) start to collect the user's voice through the microphone. In response to other applications (e.g., recording applications, phone applications, audio and video call applications, etc.) starting to collect the user's voice through the microphone, the Visible and Talking function is temporarily unavailable, and the Visible and Talking function temporarily stops occupying the microphone; the terminal device prompts the user that the Visible and Talking function is temporarily unavailable. In one implementation, when the Visible and Talking function is converted from being turned on to being temporarily unavailable, the status bar of the display interface of the terminal device no longer displays the first prompt icon, and switches to displaying the second prompt icon, and the second prompt icon indicates that the Visible and Talking function is temporarily unavailable.

[0131] In one example, if Figure 5 As shown in (a), the mobile phone 100 has turned on the see and speak function. Figure 2 The method shown turns on the "see and speak" switch on the mobile phone 100, and turns on the "see and speak" function. The mobile phone 100 exits the setting application and displays the desktop interface 120. The status bar of the display interface of the mobile phone 100 displays a prompt icon 10a, indicating that the "see and speak" function is turned on. Optionally, the "see and speak" function turns on the recording channel, and the status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b to prompt the user that the microphone is collecting voice.

[0132] Mobile phone 100 receives a video call request from another device, such as Figure 5 As shown in (b), the mobile phone 100 displays a user interface 124. Exemplarily, the user interface 124 includes a prompt message "XX invites you to make a video call" to prompt the user that the mobile phone 100 has received a video call request. The user interface 124 also includes an "answer" button 125, and the user can click the "answer" button 125 to start the video call.

[0133] The mobile phone 100 detects that the user clicks the "answer" button 125. In response to the user clicking the "answer" button 125, the mobile phone 100 starts the video call and displays Figure 5 The user interface 126 is shown in (c).

[0134] The mobile phone 100 starts a video call and collects the user's voice through the microphone; the "see and talk" function is temporarily unavailable, and the "see and talk" function no longer occupies the microphone. At this time, the preset switch corresponding to "see and talk" is still on. For example, Figure 5 As shown in (c), the status bar of the display interface of the mobile phone 100 switches from displaying the prompt icon 10a to displaying the prompt icon 10c, and the prompt icon 10c indicates that the visible and speak function is temporarily unavailable. Optionally, the mobile phone 100 also displays a prompt message 127 to prompt the user that the microphone is occupied by other applications and the voice control (visible and speak) function is temporarily unavailable. In one implementation, the prompt message 127 stays on the display interface for a preset length of time (for example, 2 seconds) and is no longer displayed. The visible and speak function temporarily stops occupying the microphone, but the video call application still collects the user's voice through the microphone. Exemplarily, the status bar of the display interface of the mobile phone 100 still displays the prompt icon 10b to prompt the user that the microphone is collecting voice.

[0135] In this example, when the See-and-Speak feature collects the user's voice through the microphone, the video call application seizes the microphone, and the See-and-Speak feature temporarily stops occupying the microphone. In other words, the See-and-Speak feature conflicts with the concurrent recording of voice by the video call application. When the video call application collects the user's voice through the microphone, the See-and-Speak feature is suspended.

[0136] In another example, Figure 6 As shown in (a), the mobile phone 100 has turned on the see and speak function. Figure 2The method shown turns on the "see and speak" switch on the mobile phone 100, and turns on the "see and speak" function. The mobile phone 100 exits the setting application and displays the desktop interface 120. The status bar of the display interface of the mobile phone 100 displays a prompt icon 10a, indicating that the "see and speak" function is turned on. Optionally, the "see and speak" function turns on the microphone, and the status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b to prompt the user that the microphone is collecting voice.

[0137] The desktop interface 120 includes an icon "recorder" for a recording application. The user can click on the icon "recorder" to start the recording application. Exemplarily, in response to receiving a click operation on the icon "recorder" by the user, the mobile phone 100 starts the recording application. Figure 6 As shown in (b), the mobile phone 100 displays a recorder interface 128. The recorder interface 128 includes a start button 1281, and the user can click the start button 1281 to start the recording function of the recorder.

[0138] Exemplarily, in response to receiving a user click operation on the start button 1281, the mobile phone 100 displays the recording interface 129. Optionally, the status bar of the display interface of the mobile phone 100 also displays a recording icon 10d, indicating that the recording application is recording voice. In other words, the recording application is collecting user voice through the microphone.

[0139] The recording application of the mobile phone 100 collects the user's voice through the microphone; in response to the recording application starting to collect the user's voice through the microphone, the see-and-speak function is temporarily stopped, and the see-and-speak function temporarily stops occupying the microphone. Figure 6 As shown in (c), the status bar of the display interface of the mobile phone 100 displays a prompt icon 10c, and the prompt icon 10c indicates that the visible and speak function is temporarily unavailable. Optionally, the recording interface 129 of the mobile phone 100 also displays a prompt message 127 to prompt the user that the microphone is occupied by other applications and the voice control (visible and speak) function is temporarily unavailable. In one implementation, the prompt message 127 stays on the display interface for a preset length of time (such as 2 seconds) and is no longer displayed. The visible and speak function temporarily stops occupying the microphone, but the recording application still collects the user's voice through the microphone. Exemplarily, the status bar of the display interface of the mobile phone 100 still displays a prompt icon 10b to prompt the user that the microphone is collecting voice.

[0140] In this example, when the See-and-Speak feature collects user voice through the microphone, the recording application seizes the microphone, and the See-and-Speak feature temporarily stops occupying the microphone. In other words, the See-and-Speak feature conflicts with the recording application's voice recording concurrently, and when the recording application collects user voice through the microphone, the See-and-Speak feature is suspended.

[0141] In some embodiments, when receiving an operation of turning on a preset switch by a user to enable the see-and-speak function, if there are other applications (e.g., a recording application, a phone application, an audio and video call application, etc.) that are collecting user voice through a microphone, in response to the user turning on the preset switch, a switch on the display interface of the terminal device is turned on, and the see-and-speak function is temporarily not enabled, and the see-and-speak function is temporarily unavailable. In one example, the display interface of the terminal device displays a second prompt icon, indicating that the see-and-speak function is temporarily unavailable.

[0142] For example, Figure 7 As shown in (a), the user records through the recording application of the mobile phone 100, and the mobile phone 100 displays the recording interface 129, and the recording application collects the user's voice through the microphone. Exemplarily, the status bar of the display interface of the mobile phone 100 displays a prompt icon 10b, which is used to prompt the user that the microphone is collecting voice. The status bar of the display interface of the mobile phone 100 also displays a recording icon 10d, indicating that the recording application is recording voice. In other words, the recording application is collecting the user's voice through the microphone.

[0143] In one example, when the recording application records voice through a microphone, a user turns on a preset switch corresponding to "see and speak". Figure 7 In (a), the mobile phone 100 receives a left swipe operation by the user on the recording interface 129. In response to the left swipe operation, the recording application exits the foreground and returns to the background. Figure 7 As shown in (b), the mobile phone 100 displays a desktop interface, and the status bar of the mobile phone 100 displays a recording icon 10d, indicating that the recording application is recording voice. The user can open the settings function of the mobile phone 100; for example, the user clicks the application icon of the "settings" application on the desktop. Figure 7 As shown in (c), in response to the user clicking the application icon of the "settings" application, the mobile phone 100 displays the "settings" interface 101. The "settings" interface 101 includes a "smart voice" option 102, and the "smart voice" option 102 is used to set the smart voice function. Figure 7 As shown in (d), in response to the user clicking the "Smart Voice" option 102, the mobile phone 100 displays the "Smart Voice" interface 103, and the "Smart Voice" interface 103 includes the "See and Speak" option 104. The user can click the "See and Speak" option 104 to set options related to the See and Speak function. Figure 7(e), in response to the user clicking the “see and say” option 104, the mobile phone 100 displays the “see and say” interface 105. The “see and say” interface 105 includes a “see and say” switch 107. The user can click the “see and say” switch 107 to turn on or off the “see and say” switch.

[0144] In one example, reference Figure 7 (f), in response to receiving a user's click operation on the "See and Speak" switch 107, the "See and Speak" switch 107 of the mobile phone 100 is turned on. At this time, the recording application is collecting the user's voice through the microphone, and the status bar of the display interface of the mobile phone 100 displays the recording icon 10d, and the See and Speak function cannot be turned on temporarily, and the See and Speak function is temporarily unavailable. Optionally, the "See and Speak" interface 105 displays a prompt message 109 to prompt the user that the See and Speak function is temporarily unavailable. In one example, the status bar of the display interface of the mobile phone 100 displays a prompt icon 10c, indicating that the See and Speak function is temporarily unavailable.

[0145] In this scenario, after the "See and Speak" switch is turned on, the See and Speak function is temporarily unavailable. Figure 8 , the mobile phone 100 displays the "See and Speak" interface 105, and the "See and Speak" switch 107 in the "See and Speak" interface 105 is turned on. The recording application is running in the background, and the status bar of the display interface of the mobile phone 100 displays the recording icon 10d, indicating that the recording application is recording voice. The status bar of the display interface of the mobile phone 100 also displays a prompt icon 10c, indicating that the "See and Speak" function is temporarily unavailable. Optionally, the status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b, which is used to prompt the user that the microphone is collecting voice.

[0146] In one example, the user performs a "swipe up" gesture at the bottom of the display screen of the mobile phone 100. In response to receiving the user's "swipe up" gesture, the mobile phone 100 exits the application interface of the "settings" application and displays the desktop interface 120. In this display interface, the status bar displays a recording icon 10d, indicating that the recording application is recording voice; the status bar also displays a prompt icon 10c, indicating that the see-and-speak function is temporarily unavailable. Optionally, the status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b to prompt the user that the microphone is collecting voice.

[0147] In some embodiments, the "See and Speak" switch is turned on. Since other applications (such as recording applications, phone applications, audio and video call applications, etc.) collect user voice through the microphone, the "See and Speak" function is temporarily unavailable. When other applications occupying the microphone no longer occupy the microphone, the "See and Speak" function is automatically turned on; the "See and Speak" function collects user voice through the microphone. In one implementation, the "See and Speak" function is automatically turned on, and the terminal device no longer displays the second prompt icon, and switches to displaying the first prompt icon.

[0148] In one example, if Fig. 9 As shown in (a), the user uses the mobile phone 100 to make a video call, and the mobile phone 100 displays a user interface 126. The video call application collects the user's voice through the microphone, and the see-and-speak function is temporarily unavailable. The status bar of the display interface of the mobile phone 100 displays a prompt icon 10c, and the prompt icon 10c indicates that the see-and-speak function is temporarily unavailable. The status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b to prompt the user that the microphone is collecting voice.

[0149] The Visible and Talk function is temporarily unavailable, which means that the Visible and Talk switch is turned on, but the microphone is occupied by other applications. When the microphone is no longer occupied by other applications, the terminal device automatically turns on the Visible and Talk function.

[0150] Exemplarily, the user interface 126 includes a "hang up" button, and the user can click the "hang up" button to stop the video call. Fig. 9 As shown in (b), in response to the user clicking the “hang up” button, the mobile phone 100 closes the video call application and displays the desktop interface 120.

[0151] After the video call application is closed, the microphone is no longer occupied; the visible and speaking function is successfully turned on. Fig. 9 As shown in (b), the status bar of the display interface of the mobile phone 100 no longer displays the prompt icon 10c; the status bar displays the prompt icon 10a, indicating that the visible and speaking function is turned on. The visible and speaking function collects the user's voice through the microphone, and the status bar of the display interface of the mobile phone 100 also displays the prompt icon 10b to prompt the user that the microphone is collecting voice. Optionally, the mobile phone 100 also displays a prompt message 130 to prompt the user that the visible and speaking function is turned on.

[0152] In this example, the see-and-talk feature is temporarily unavailable when the video call application is occupying the microphone. When the video call application is no longer occupying the microphone, the see-and-talk feature is automatically enabled.

[0153] In another example, Fig.10As shown in (a), the user uses the mobile phone 100 to record, and the mobile phone 100 displays the recording interface 129. The recording application records the voice through the microphone, and the visible and speaking function is temporarily unavailable. Exemplarily, the status bar of the display interface of the mobile phone 100 displays the recording icon 10d, indicating that the recording application is recording the voice. The status bar of the display interface of the mobile phone 100 also displays the prompt icon 10c, indicating that the visible and speaking function is temporarily unavailable. Optionally, the status bar of the display interface of the mobile phone 100 also displays the prompt icon 10b to prompt the user that the microphone is collecting voice.

[0154] The Visible and Talk function is temporarily unavailable, which means that the Visible and Talk switch is turned on, but the microphone is occupied by other applications. When the microphone is no longer occupied by other applications, the terminal device automatically turns on the Visible and Talk function.

[0155] Exemplarily, the recording interface 129 includes a "stop" button, and the user can click the "stop" button to stop recording the voice. Fig.10 As shown in (b), in response to the user clicking the “Stop” button, the recording application stops recording voice, and the mobile phone 100 displays the recorder interface 128 .

[0156] After the recording application stops recording voice through the microphone, the microphone is no longer occupied; it can be seen that the speak function is successfully turned on. Fig.10 As shown in (b), the recording application no longer occupies the microphone, and the status bar of the display interface of the mobile phone 100 no longer displays the recording icon 10d; the visible and speaking function is turned on, and the status bar of the display interface of the mobile phone 100 no longer displays the prompt icon 10c; the status bar displays the prompt icon 10a, indicating that the visible and speaking function is turned on. The visible and speaking function collects user voice through the microphone, and the status bar of the display interface of the mobile phone 100 also displays the prompt icon 10b to remind the user that the microphone is collecting voice. Optionally, the mobile phone 100 also displays a prompt message 130 to remind the user that the visible and speaking function is turned on.

[0157] In this example, when the recording application occupies the microphone, the see-and-speak feature is temporarily unavailable. When the recording application no longer occupies the microphone, the see-and-speak feature is automatically enabled.

[0158] On the terminal device, the voice control function needs to collect the user's voice through the microphone, so that the corresponding instructions can be executed according to the user's voice; if there are other applications or functions (such as recording applications, telephone applications, audio and video call applications, etc.), they also need to record voice through the microphone; the voice control function and other applications recording voice will cause concurrency conflicts. The method for handling concurrency conflicts of recorded voice provided in the embodiment of the present application, when the switch of the voice control function is turned on, if there are other applications (or functions) occupying the microphone to record voice, the voice control function will be temporarily unavailable. In this way, it is avoided that the voice control function mistakenly executes the instructions corresponding to the voice, causing misoperation.

[0159] In the method for handling concurrent conflicts of recorded voice provided in the embodiment of the present application, when the preset switch corresponding to the voice control function is not turned on, the voice control function is turned off. When the preset switch corresponding to the voice control function is turned on, if the voice control function is temporarily unavailable, the status bar of the terminal device display interface displays a second prompt icon (e.g., prompt icon 10c). When the preset switch corresponding to the voice control function is turned on, if the voice control function is successfully turned on, the status bar of the terminal device display interface displays a first prompt icon (e.g., prompt icon 10a). In this way, the user can easily know whether the voice control function is successfully turned on through the prompt icon.

[0160] Audio Driver Reference Figure 5 and Figure 6 The illustrated scenario example, in some embodiments, when the See and Speak feature is turned on and the user's voice is collected through the microphone, other applications (such as recording applications, phone applications, audio and video call applications, etc.) request to collect the user's voice through the microphone. In response to the other applications (such as recording applications, phone applications, audio and video call applications, etc.) starting to collect the user's voice through the microphone, the See and Speak feature temporarily stops occupying the microphone.

[0161] Combination Figure 4A The software architecture of the terminal device 100 is shown in FIG. Fig.11 A flow chart of a method for handling concurrent conflicts in recorded speech provided by an embodiment of the present application in this scenario is shown.

[0162] like Fig.11 As shown, the method for processing concurrent conflicts of recorded voices provided in the embodiment of the present application may include:

[0163] S1101: After receiving an operation from a user to turn on a see-and-speak function switch, the recording management module queries the recording module for microphone occupancy status.

[0164] refer to Figure 4B-4DEach application requests ADSP to establish a recording channel through the recording module. The recording module can manage the establishment status of the recording channel. If there is a recording channel corresponding to a certain application, it means that the microphone is occupied.

[0165] The recording management module queries the recording module for the microphone occupancy status, which includes occupied and unoccupied.

[0166] S1102. The recording module returns to the recording management module that the microphone is not occupied.

[0167] S1103: In response to the user turning on the "see and speak" function switch, the recording management module turns on the "see and speak" function.

[0168] In one example, the user can enable the see-and-speak function by turning on the see-and-speak function switch. Figure 2 The method shown turns on the see-and-speak switch on the mobile phone 100. The recording management module determines that the microphone occupancy state is unoccupied, and in response to the user's operation of turning on the see-and-speak function switch, turns on the see-and-speak function.

[0169] S1104. The recording management module notifies the recording module to establish a recording channel corresponding to the see-and-speak function.

[0170] In one implementation, the recording management module notifies the recording module to establish a recording channel (AudioRecord) corresponding to the see-and-speak function.

[0171] S1105. The recording module notifies the ADSP to establish a recording channel.

[0172] The recording module notifies ADSP to turn on the microphone and collect audio streams. ADSP establishes a recording channel (AudioRecord) corresponding to the See-and-Speak function, and the recording module distributes the audio stream collected by the microphone to the See-and-Speak application.

[0173] S1106. The recording management module notifies the status management module to record that the status of the see-and-speak function is turned on.

[0174] Exemplarily, the status of the see-and-tell function may include turned on, turned off, temporarily unavailable, etc.

[0175] S1107. The recording management module also notifies the monitoring module to monitor the occupancy status of the microphone.

[0176] S1108. The monitoring module registers monitoring with the recording module.

[0177] In one implementation, if the application needs to obtain an audio stream, it needs to request the recording module to establish an AudioRecord corresponding to the application. The recording module can determine the occupancy status of the microphone according to the started AudioRecord; that is, the recording module can monitor the occupancy status of the microphone.

[0178] S1109. Other applications request to record voice through the microphone.

[0179] For example, refer to Figure 5 In the illustrated scenario, the video call application requests to record voice through a microphone. In one implementation, the video call application requests the recording module to create an AudioRecord corresponding to the video call application.

[0180] For example, refer to Figure 6 In the illustrated scenario, the recording application requests to record voice through a microphone. In one implementation, the recording application requests the recording module to create an AudioRecord corresponding to the recording application.

[0181] S1110. The recording module notifies the monitoring module that the microphone is occupied by other applications.

[0182] S1111. The monitoring module notifies the recording management module that the microphone is occupied by other applications.

[0183] S1112. The recording management module notifies the recording module to close the recording channel corresponding to the see-and-speak function.

[0184] When the recording management module receives a notification that the microphone is occupied by other applications, it temporarily stops collecting audio streams through the microphone. In one implementation, the recording management module notifies the recording module to close the AudioRecord corresponding to the Visible and Talk function. In this way, the audio stream collected by the microphone will no longer be distributed to the Visible and Talk application.

[0185] S1113: The recording module notifies ADSP to close the previous recording channel and re-establish a recording channel.

[0186] ADSP establishes a recording channel corresponding to other applications, and the audio stream collected by the microphone will be distributed to the application.

[0187] S1114. The recording management module notifies the status management module to update the status of the see-and-speak function to temporarily unavailable, and notifies the service presentation module to issue a prompt on the user interface that the see-and-speak function is temporarily unavailable.

[0188] S1115. The service presentation module issues a prompt message on the user interface indicating that the visible-to-speak function is temporarily unavailable, and displays a prompt icon indicating that the visible-to-speak function is temporarily unavailable.

[0189] In one implementation, the service presentation module issues a prompt on the user interface to prompt the user that the visible function is temporarily unavailable. Figure 5 (c) and Figure 6 As shown in (c), the status bar of the display interface of the mobile phone 100 displays a prompt icon 10c, indicating that the see-and-speak function is temporarily unavailable. The mobile phone 100 also displays a prompt message 127 to prompt the user that the microphone is occupied by other applications and the see-and-speak function is temporarily unavailable.

[0190] refer to Figure 7 The scenario example shown, in some embodiments, when the user turns on the preset switch to enable the See-and-Speak function, there are other applications (for example, a recording application, a phone application, an audio and video call application, etc.) that are collecting the user's voice through the microphone, and the terminal device temporarily does not enable the See-and-Speak function, and the See-and-Speak function is temporarily unavailable.

[0191] Combination Figure 4A The software architecture of the terminal device 100 is shown in FIG. Fig.12 A flow chart of a method for handling concurrent conflicts in recorded speech provided by an embodiment of the present application in this scenario is shown.

[0192] like Fig.12 As shown, the method for processing concurrent conflicts of recorded voices provided in the embodiment of the present application may include:

[0193] S1201: After receiving an operation from a user to turn on a see-and-speak function switch, the recording management module queries the recording module for microphone occupancy status.

[0194] S1202: The recording module returns information that the microphone is occupied to the recording management module.

[0195] In one implementation, when each application requests to collect an audio stream through a microphone, it requests the recording module to establish an AudioRecord corresponding to the application. The recording module can determine the occupancy status of the microphone according to the started AudioRecord; that is, the recording module can monitor the occupancy status of the microphone.

[0196] S1203. The recording management module notifies the status management module to update the status of the visible-to-speak function to temporarily unavailable.

[0197] The recording management module receives the user's operation of turning on the "See and Speak" function switch, and queries the recording module for the microphone occupancy status. If the microphone is occupied, it means that other applications are collecting audio streams through the microphone, and the "See and Speak" function is not enabled temporarily. The recording management module notifies the status management module to update the "See and Speak" function status to temporarily unavailable.

[0198] S1204: The recording management module notifies the service presentation module to issue a prompt on the user interface that the visible-to-speak function is temporarily unavailable.

[0199] S1205. The service presentation module issues a prompt message on the user interface indicating that the visible-to-speak function is temporarily unavailable, and displays a prompt icon indicating that the visible-to-speak function is temporarily unavailable.

[0200] In one implementation, the service presentation module issues a prompt on the user interface to prompt the user that the visible function is temporarily unavailable. Figure 7 As shown in (f), the status bar of the display interface of the mobile phone 100 displays a prompt icon 10c, indicating that the see-and-speak function is temporarily unavailable. The mobile phone 100 also displays a prompt message 109 to prompt the user that the microphone is occupied by other applications and the see-and-speak function is temporarily unavailable.

[0201] S1206: The recording management module notifies the monitoring module to monitor the microphone occupancy status, and the monitoring module registers monitoring with the recording module.

[0202] The switch of the visible-and-speaking function is turned on, the recording management module notifies the monitoring module to monitor the microphone occupancy status, and the monitoring module registers monitoring with the recording module.

[0203] In one implementation, the recording module can monitor the microphone occupancy status according to the startup of AudioRecord. After the monitoring module registers monitoring with the recording module, if the microphone occupancy status changes, the recording module can notify the monitoring module. In one example, if other applications no longer occupy the microphone, the recording module can notify the monitoring module that other applications have quit occupying the microphone (there is no application occupying the microphone).

[0204] refer to Fig. 9 and Fig.10 In the example scenario shown, in some embodiments, the switch of the see-and-speak function is turned on. Because other applications (such as recording applications, phone applications, audio and video call applications, etc.) collect user voice through the microphone, the see-and-speak function is temporarily unavailable. When other applications occupying the microphone no longer occupy the microphone, the see-and-speak function is automatically turned on.

[0205] Combination Figure 4A The software architecture of the terminal device 100 is shown in FIG. Fig.13 A flow chart of a method for handling concurrent conflicts in recorded speech provided by an embodiment of the present application in this scenario is shown.

[0206] like Fig.13 As shown, the method for processing concurrent conflicts of recorded voices provided in the embodiment of the present application may include:

[0207] S1301: The recording module detects that other applications have exited and occupied the microphone.

[0208] After the switch of the See and Speak function is turned on, the monitoring module registers the monitoring with the recording module. In this way, when the recording module detects a change in the microphone occupancy status, it can notify the monitoring module.

[0209] In one example, the recording module receives a request from other applications to close the recording channel, that is, it monitors that other applications exit and occupy the microphone.

[0210] Optionally, the recording module notifies ADSP to close the recording channel corresponding to the application.

[0211] S1302: The recording module notifies the monitoring module that other applications have exited and occupied the microphone.

[0212] S1303: The monitoring module notifies the recording management module that other applications have exited and occupied the microphone.

[0213] The recording management module receives a notification that other applications have exited and occupied the microphone, and determines that the switch of the visible-to-speak function has been turned on (confirming through the status management module that the status of the visible-to-speak function is temporarily unavailable), and then turns on the visible-to-speak function.

[0214] S1304. The recording management module notifies the recording module to establish a recording channel corresponding to the see-and-speak function.

[0215] In one implementation, the recording management module notifies the recording module to create an AudioRecord corresponding to the see-and-speak function.

[0216] S1305. The recording module notifies the ADSP to establish a recording channel corresponding to the see-and-speak function.

[0217] In this way, the audio stream collected by the microphone will be distributed to the See and Speak application.

[0218] S1306. The recording management module notifies the status management module to update the status of the visible-to-speak function to enabled, and notifies the service presentation module to issue a prompt on the user interface that the visible-to-speak function is enabled.

[0219] S1307. The business presentation module sends a prompt message on the user interface indicating that the visible-to-speak function has been turned on, and displays a prompt icon indicating that the visible-to-speak function has been turned on.

[0220] In one implementation, the service presentation module issues a prompt on the user interface to prompt the user that the visible function has been turned on. Fig. 9 (b) and Fig.10As shown in (b), the status bar of the display interface of the mobile phone 100 no longer displays the prompt icon 10c; the status bar of the display interface of the mobile phone 100 displays the prompt icon 10a, indicating that the visible and speaking function is turned on. The mobile phone 100 also displays a prompt message 130 to prompt the user that the visible and speaking function is turned on.

[0221] The method for handling concurrent conflicts in recorded voice provided in an embodiment of the present application, when there are recording channels of other applications, it means that there are other applications occupying the microphone, and the Visible and Speak function is temporarily unavailable. In some embodiments, when the user's operation of turning on the Visible and Speak function switch is received, it is determined that there are recording channels corresponding to other applications, and the Visible and Speak function is temporarily not turned on. In some embodiments, the Visible and Speak function is turned on, and other applications request to establish corresponding recording channels, then the corresponding recording channels of Visible and Speak are closed. In this way, the audio stream collected by the microphone will be distributed to the application through the recording channels of other applications, and will not be distributed to the Visible and Speak application. Visible and Speak is avoided from being triggered by mistake.

[0222] When other applications exit and occupy the microphone (the corresponding recording channels of other applications are closed), the Visible and Talk function is automatically started and the corresponding recording channel of Visible and Talk is established. In this way, the audio stream collected by the microphone will be distributed to the Visible and Talk application. Users can control the terminal device through voice.

[0223] In the method for handling concurrent conflicts of recorded voice provided in the embodiment of the present application, after the switch of the see-and-speak function is turned on, the see-and-speak application monitors the occupancy status of the microphone (the situation of establishing a recording channel), and controls the see-and-speak function to be turned on or temporarily unavailable according to the occupancy status of the microphone. There is no need to modify the processing flow of other applications or operating systems, and the compatibility is strong.

[0224] Below, taking the other application as a voice assistant application as an example, the method for handling concurrent conflicts in recorded voice provided in the embodiment of the present application is exemplarily introduced. Voice assistant and see and speak are two voice control functions. In the embodiment of the present application, see and speak is recorded as the first voice control function, and the voice assistant is recorded as the second voice control function.

[0225] In some embodiments, after the Visible and Talking function is turned on, in response to a user operation, the terminal device is awakened and the voice assistant function is turned on. The voice assistant function occupies the microphone, and the Visible and Talking function is temporarily unavailable. After the voice assistant function is turned on, the user can issue a command to the terminal device by inputting a voice into the terminal device, and the terminal device executes the command corresponding to the voice. After the terminal device executes a command, or the terminal device does not receive a command issued by the user through voice within a period of time (for example, within 8 seconds) after being awakened, the voice assistant function is turned off. The voice assistant function no longer occupies the microphone, and the Visible and Talking function is automatically turned on.

[0226] Exemplary, reference Fig.14 (a), the mobile phone 100 has turned on the see-and-speak function. Figure 2 The method shown turns on the "see and speak" switch on the mobile phone 100, and turns on the "see and speak" function. The mobile phone 100 exits the setting application and displays the desktop interface 120. The status bar of the display interface of the mobile phone 100 displays a prompt icon 10a, indicating that the "see and speak" function is turned on. Optionally, the "see and speak" function turns on the microphone, and the status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b to prompt the user that the microphone is collecting voice.

[0227] The user long presses the power button of the mobile phone 100. In response to receiving the long press of the power button by the user, the mobile phone 100 is awakened. After the mobile phone 100 is awakened, a prompt card is displayed, indicating that the mobile phone 100 has been awakened and the user can control the mobile phone by voice. The voice assistant function occupies the microphone, which means that the voice assistant function is temporarily unavailable. For example, Fig.14 As shown in (b), the mobile phone 100 displays a voice assistant prompt card, indicating that the mobile phone 100 has been awakened, which is used to prompt the user to input voice into the mobile phone 100 to control the mobile phone 100. The status bar of the display interface of the mobile phone 100 switches from displaying the prompt icon 10a to displaying the prompt icon 10c, indicating that the visible and speaking function is temporarily unavailable.

[0228] For example, Fig.14 As shown in (c), the user inputs the voice "open video" into the mobile phone 100, and the mobile phone 100 analyzes and recognizes the user's voice and executes the instruction corresponding to the voice "open video".

[0229] For example, Fig.14 As shown in (d), the mobile phone starts the video application in response to receiving the user's voice "open video". After the mobile phone 100 executes the instruction, the voice assistant function is turned off. The voice assistant function no longer occupies the microphone, and the see-and-speak function is automatically turned on. The status bar of the display interface of the mobile phone 100 no longer displays the prompt icon 10c; the status bar displays the prompt icon 10a, indicating that the see-and-speak function is turned on.

[0230] Combination Figure 4A The software architecture shown, Fig.15 It shows the interaction process of each module when the voice assistant function and the see-and-speak function are concurrent.

[0231] Exemplary, reference Fig.15 After the Visible and Talking function is enabled, a recording channel (recording channel 1) corresponding to the Visible and Talking function is established. The Visible and Talking application also registers a listener with the recording module.

[0232] S1. The user long presses the power button of the mobile phone. The voice assistant application receives the user's long press of the power button operation.

[0233] S2. In response to receiving an operation of long pressing the power button by the user, the voice assistant application requests the recording module to establish a recording channel corresponding to the voice assistant function.

[0234] S3. If the recording module notification is visible, it means that the application microphone is occupied by other applications.

[0235] S4. The visible and visible application notifies the recording module to close the recording channel 1.

[0236] S5. The recording module notifies ADSP to close recording channel 1 and requests ADSP to re-establish a recording channel.

[0237] ADSP establishes recording channel 2, which is the recording channel corresponding to the voice assistant application.

[0238] Furthermore, after the mobile phone executes a voice command, the voice assistant function is turned off. The voice assistant application requests the recording module to turn off the recording channel corresponding to the voice assistant function. The recording module notifies the ADSP to turn off recording channel 2. The recording module notifies the Visible and Talk application that other applications exit the microphone occupation. The Visible and Talk application notifies the recording module to establish the recording channel corresponding to the Visible and Talk function.

[0239] In this embodiment, after the Visible and Talking function is activated, a recording channel corresponding to the Visible and Talking function is established. After the voice assistant function is activated, the voice assistant function occupies the microphone. The recording channel corresponding to the Visible and Talking function is closed, and a recording channel corresponding to the voice assistant function is established. After the voice assistant function is closed, the recording channel corresponding to the voice assistant function is closed, and a recording channel corresponding to the Visible and Talking function is established again. At the same time, there is only one recording channel, and the audio stream collected by the microphone will be distributed to the corresponding application. This can avoid the system processing errors caused by both the Visible and Talking function and the Voice Assistant function being turned on.

[0240] In some embodiments, when there is a recording channel, the ADSP in the kernel layer of the terminal device notifies the wake word detection module to stop wake word detection. That is, when any application collects voice through the microphone, the terminal device stops wake word detection. Exemplarily, the visible speech function of the mobile phone 100 has been enabled, and the mobile phone 100 displays Fig.14 the desktop interface 120 shown in (a) of. The status bar of the display interface of the mobile phone 100 displays a prompt icon 10a, indicating that the visible speech function has been enabled. Optionally, when the visible speech function enables the microphone, the status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b, which is used to prompt the user that the microphone is collecting voice. The user inputs the voice "Hello YOYO" to the mobile phone 100, and the mobile phone 100 will not be woken up.

[0241] Exemplarily, Fig.16 is a schematic structural diagram of the terminal device 100. Among them, the terminal device 100 may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone interface 270D, a sensor module 280, a key 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. Among them, the sensor module 280 may include a temperature sensor, an ambient light sensor, a touch sensor, etc.

[0242] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device 100. In other embodiments, the terminal device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0243] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0244] The controller may be the nerve center and command center of the terminal device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0245] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0246] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0247] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation on the terminal device. In other embodiments, the terminal device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0248] The charging management module 240 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 may receive charging input from a wired charger through the USB interface 230. In some wireless charging embodiments, the charging management module 240 may receive wireless charging input through a wireless charging coil of a terminal device. While the charging management module 240 is charging the battery 242, it may also power the terminal device through the power management module 241.

[0249] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 241 can also be set in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

[0250] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0251] The terminal device 100 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0252] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-OLED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc.

[0253] The terminal device 100 can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0254] The ISP is used to process the data fed back by the camera 293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

[0255] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0256] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0257] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. In this way, the terminal device can play or record videos in multiple coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0258] NPU is a neural network (NN) computing processor. It can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of terminal devices, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0259] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the earphone interface 270D, and the application processor.

[0260] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 are arranged in the processor 210. The speaker 270A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 270B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. The microphone 270C, also known as the "microphone" and the "microphone", is used to convert sound signals into electrical signals. The headphone interface 270D is used to connect wired headphones. The headphone interface 270D can be a USB interface 230, or it can be a 3.5mm open mobile electronic device platform (open mobile terminal platform, OMTP) standard interface, the cellular telecommunications industry association of the USA (cellular telecommunications industry association of the USA, CTIA) standard interface. In the embodiment of the present application, after the voice control function is turned on, the terminal device 100 continuously collects surrounding sounds through the microphone 270C.

[0261] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, audio, video and other files are stored in the external memory card.

[0262] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the terminal device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 can execute instructions stored in the internal memory 221, and the internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store data (such as video files) created during the use of the terminal device, etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0263] The button 290 includes a power button, a volume button, etc. The button 290 can be a mechanical button. It can also be a touch button. The motor 291 can generate a vibration prompt. The motor 291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 292 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 295, or pulled out from the SIM card interface 295 to achieve contact and separation with the terminal device. The terminal device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0264] It is understandable that, in order to realize the above functions, the above terminal device includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0265] The embodiment of the present application can divide the functional modules of the above-mentioned terminal device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0266] In the case of an integrated unit, Fig.17 A possible structural diagram of a terminal device involved in the above embodiment is shown. The terminal device 1600 includes: a processing unit 1601 , a storage unit 1602 , an audio unit 1603 and a display unit 1604 .

[0267] The processing unit 1601 is used to control and manage the actions of the terminal device 1600. For example, it can be used to control the opening, temporary unavailability and closing of the voice control function; it can be used to parse and recognize the user voice when receiving the user voice, and / or used for other processes of the technology described in this article.

[0268] The storage unit 1602 is used to store the program code and data of the terminal device 1600. The processing unit 1601 calls the program code stored in the storage unit 1602 to execute each step in the above method embodiment.

[0269] The audio unit 1603 is used to collect the voice uttered by the user and play the voice.

[0270] The display unit 1604 is used to display the interface of the terminal device 1600 .

[0271] Of course, the unit modules in the terminal device 1600 include but are not limited to the processing unit 1601, the storage unit 1602, the audio unit 1603 and the display unit 1604. For example, the terminal device 1600 may also include a power supply unit, a communication unit, etc. The power supply unit is used to supply power to the terminal device 1600. The communication unit is used to support communication between the terminal device 1600 and other devices.

[0272] Among them, the processing unit 1601 can be a processor or a controller, for example, a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The processor may include an application processor and a baseband processor. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The storage unit 1602 may be a memory. The audio unit 1603 may include a microphone, a speaker, a receiver, etc. The display unit 1604 may be a display screen, etc. The communication unit may be a transceiver, a transceiver circuit or a communication interface, etc.

[0273] For example, the processing unit 1601 is a processor (such as Fig.16 The processor 210 shown in FIG. 10A ) and the storage unit 1602 may be a memory (eg, Fig.16 The audio unit 1603 may include a microphone (eg, Fig.16 Microphone 270C shown), speaker (such as Fig.16 Speaker 270A shown), receiver (such as Fig.16 The display unit 1604 may be a display screen (eg, Fig.16 The communication unit includes a mobile communication module (such as Fig.16 The mobile communication module 250 shown) and the wireless communication module (such as Fig.16 The mobile communication module and the wireless communication module may be collectively referred to as a communication interface. The terminal device 1600 provided in the embodiment of the present application may be Fig.16 The terminal device 100 shown in FIG. 1 is a schematic diagram of a terminal device 100. The processor, memory, microphone, display screen, and communication interface, etc., may be coupled together, for example, via a bus. The processor calls the program code stored in the memory to execute each step in the above method embodiment.

[0274] The present application also provides a chip system (eg, a system on a chip (SoC)), such as Fig.18As shown, the chip system includes at least one processor 1701 and at least one interface circuit 1702. The processor 1701 and the interface circuit 1702 can be interconnected via lines. For example, the interface circuit 1702 can be used to receive signals from other devices (such as a memory of a terminal device). For another example, the interface circuit 1702 can be used to send signals to other devices (such as a processor 1701 or a touch screen of a terminal device). Exemplarily, the interface circuit 1702 can read instructions stored in the memory and send the instructions to the processor 1701. When the instructions are executed by the processor 1701, the terminal device can execute the various steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0275] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned terminal device, the terminal device executes each function or step in the above-mentioned method embodiment.

[0276] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0277] Among them, the terminal device 1600, chip system, computer-readable storage medium or computer program product provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0278] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0279] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0280] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0281] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0282] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0283] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing concurrent conflicts in recorded voice, applied to a terminal device, wherein the terminal device includes a microphone, and is characterized in that: The method comprises: The terminal device displays a first interface, wherein the first interface includes a first switch; In response to the user turning on the first switch, the terminal device displays a first prompt icon, where the first prompt icon indicates that the first voice control function has been turned on; After receiving the first voice collected by the microphone, the terminal device executes a first instruction corresponding to the first voice; the first voice does not include a wake-up word of the terminal device; In response to the user's operation of turning on the second voice control function, the terminal device stops displaying the first prompt icon, displays the second prompt icon, and displays the first prompt card; the second prompt icon indicates that the first voice control function is temporarily unavailable, and the first prompt card indicates that the second voice control function is turned on; After receiving the second voice collected by the microphone, the terminal device executes the second instruction corresponding to the second voice and stops displaying the first prompt card; After executing the second instruction corresponding to the second voice, the terminal device stops displaying the second prompt icon and displays the first prompt icon.

2. The method according to claim 1, characterized in that The method further comprises: In response to the user turning on the first switch, the terminal device turns on the recording channel corresponding to the first voice control function, and the recording channel is used to obtain the user voice collected by the microphone.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: In response to the user's operation of turning on the second voice control function, the terminal device turns on the recording channel corresponding to the second voice control function and turns off the recording channel corresponding to the first voice control function.

4. The method according to any one of claims 1 to 3, characterized in that: After executing the second instruction corresponding to the second voice, the method further includes: The terminal device closes the recording channel corresponding to the second voice control function, and opens the recording channel corresponding to the first voice control function.

5. The method according to any one of claims 1 to 4, characterized in that: After the terminal device stops displaying the second prompt icon and displays the first prompt icon, the method further includes: In response to the user's operation of turning on the first function, the terminal device turns on the recording channel corresponding to the first function and turns off the recording channel corresponding to the first voice control function, stops displaying the first prompt icon and displays the second prompt icon; In response to the user's operation of closing the first function, the terminal device closes the recording channel corresponding to the first function and opens the recording channel corresponding to the first voice control function, stops displaying the second prompt icon and displays the first prompt icon.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to the user's operation of turning off the first switch, the terminal device turns off the recording channel corresponding to the first voice control function.

7. The method according to any one of claims 1 to 6, characterized in that: The operation of enabling the second voice control function includes: Long press the power button on the terminal device.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When the terminal device displays the first prompt icon, the terminal device receives a third voice collected by the microphone; the third voice includes a wake-up word of the terminal device; The terminal device continues to display the first prompt icon.

9. The method according to claim 8, characterized in that After the terminal device receives the third voice collected by the microphone, the method further includes: The terminal device obtains the user voice through the recording channel corresponding to the first voice control function.

10. A method for processing concurrent conflicts in recorded voice, applied to a terminal device, the terminal device comprising a microphone, characterized in that: The method comprises: The terminal device displays a first interface, wherein the first interface includes a first switch; In response to the user turning on the first switch, the terminal device turns on the recording channel corresponding to the voice control function and displays a first prompt icon; the recording channel is used to obtain the user voice collected by the microphone, and the first prompt icon indicates that the voice control function is turned on; After receiving the first voice collected by the microphone, the terminal device executes a first instruction corresponding to the first voice; In response to the user's first operation, the terminal device establishes a recording channel corresponding to the first application and closes the recording channel corresponding to the voice control function, stops displaying the first prompt icon and displays a second prompt icon; the second prompt icon indicates that the voice control function is temporarily unavailable.

11. The method according to claim 10, characterized in that The method further comprises: In response to a second operation by the user, the terminal device closes the recording channel corresponding to the first application; In response to the recording channel corresponding to the first application being closed, the terminal device turns on the recording channel corresponding to the voice control function, stops displaying the second prompt icon and displays the first prompt icon.

12. A method for processing concurrent conflicts in recorded speech, applied to a terminal device, the terminal device comprising a microphone, characterized in that: The method comprises: In response to a first operation of the user, the terminal device establishes a recording channel corresponding to the first application; the recording channel is used to obtain the user voice collected by the microphone; The terminal device displays a first interface, wherein the first interface includes a first switch; the first switch is a switch corresponding to a voice control function; receiving an operation of turning on the first switch by a user; In response to the user's operation of turning on the first switch, the terminal device displays a prompt icon, where the prompt icon indicates that the voice control function is temporarily unavailable.

13. The method according to claim 12, characterized in that The method further comprises: In response to a second operation by the user, the terminal device closes the recording channel corresponding to the first application; In response to the recording channel corresponding to the first application being closed, the terminal device opens the recording channel corresponding to the voice control function; After receiving the first voice collected by the microphone, the terminal device executes the first instruction corresponding to the first voice.

14. A terminal device, characterized in that: The terminal device includes: a processor, a memory and a microphone, the microphone is used to collect user voice, and the processor is coupled to the memory; the memory is used to store computer program code; the computer program code includes computer instructions, and when the processor executes the above-mentioned computer instructions, the terminal device executes the method described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 13.

Images