Method for processing voice-recording concurrence conflict, and terminal device

Abstract

The present application relates to the technical field of terminals. Disclosed are a method for processing a voice-recording concurrence conflict, and a terminal device. After a first voice control function has been enabled, in response to a user operation, a terminal device is woken up, and a second voice control function is enabled. After the second voice control function is enabled, the first voice control function is temporarily unavailable. If the terminal device has executed a voice instruction by means of the second voice control function, or if the terminal device has not received, within a period of time after same is woken up, an instruction that a user issues by means of voice, the second voice control function is disabled. After the second voice control function is disabled, the first voice control function is automatically enabled. That is to say, when another application collects voice of the user by means of a microphone, the first voice control function is temporarily unavailable, thereby avoiding the problem of normal use of the terminal device performed by the user being affected due to false triggering of a first voice control.

Description

Method and terminal device for handling concurrent conflicts in voice recording

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 30, 2023, with application number 202310802734.0 and invention name “Method and terminal device for handling concurrent conflicts in recorded voice”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a method for processing concurrent conflicts in recorded voice and a terminal device. Background Art

[0003] With the advancement of terminal technology, terminal devices are becoming increasingly common in our daily lives. To improve user efficiency, these devices typically offer a variety of human-computer interaction methods, including a human-computer interface (HMI), voice control, and gesture control. Voice control is a primary and convenient method of HMI. This is particularly useful when users are driving, cooking, reading, or using their hands to control their devices.

[0004] When voice control is enabled on a terminal device, the microphone collects user voice. Several applications or features on the terminal device, such as recording apps, phone apps, and audio and video call apps, also record voice through the microphone. If these applications or features simultaneously record voice, concurrency conflicts may occur. Managing concurrency conflicts among multiple applications or features is a critical issue.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method and terminal device for handling concurrent conflicts in voice recording, which can resolve concurrent conflicts in voice recording for multiple applications or functions, avoid abnormal processing by the terminal device, and improve the user experience of using the voice control function.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a method for handling concurrent voice recording conflicts is provided, the method comprising: a terminal device displaying a first switch of a first interface; in response to a user turning on the first switch, the terminal device displaying a first prompt icon indicating that a first voice control function is enabled; after the first voice control function is enabled, the user can control the terminal device through voice; for example, upon receiving a first voice captured by a microphone, the terminal device executes a first instruction corresponding to the first voice. The first voice does not include a wake-up word for the terminal device; that is, the first voice control function is not a voice assistant, for example, the first voice control function is visible and can be spoken.

[0009] When the first voice control function is already enabled, in response to the user's operation to enable the second voice control function, the terminal device stops displaying the first prompt icon and displays the second prompt icon and the first prompt card; the second prompt icon indicates that the first voice control function is temporarily unavailable, and the first prompt card indicates that the second voice control function is enabled. In other words, in response to the second voice control function being enabled, the first voice control function is temporarily unavailable.

[0010] When the second voice control function is turned on, the user can control the terminal device through voice; for example, after receiving the second voice collected by the microphone, the terminal device executes the second instruction corresponding to the second voice; after the terminal device executes the second instruction corresponding to the second voice, it turns off the second voice control function and stops displaying the first prompt card.

[0011] After executing the second instruction corresponding to the second voice, the second voice control function is turned off and the first voice control function is automatically turned on; the terminal device stops displaying the second prompt icon and displays the first prompt icon, indicating that the first voice control function is turned on.

[0012] In this method, after the "See and Speak" function (first voice control function) is turned on, in response to a user operation, the terminal device is awakened and the voice assistant function (second voice control function) is turned on. The voice assistant function occupies the microphone, and the "See and Speak" function is temporarily unavailable. After the voice assistant function is turned on, the user can give a command to the terminal device by inputting a voice into the terminal device, and the terminal device executes the command corresponding to the voice. After the terminal device executes a command, or if the terminal device does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after being awakened, the voice assistant function is turned off. The voice assistant function no longer occupies the microphone, and the "See and Speak" function is automatically turned on.

[0013] Once the first voice control function is enabled, it continuously collects user voice through the microphone. If another application mistakenly distributes the user's voice to the first voice control function while collecting the user's voice through the microphone, the first voice control function will be mistakenly triggered to execute the corresponding voice command. In this method, when other applications (such as the second voice control function) collect the user's voice through the microphone, the visible and speak function (the first voice control function) is suspended (without occupying the microphone). This can prevent the first voice control function from being mistakenly triggered to execute voice commands and affecting the user's normal use of the terminal device.

[0014] In combination with the first aspect, in a possible implementation, in response to the user turning on the first switch, the terminal device turns on the recording channel corresponding to the first voice control function, and the recording channel is used to obtain the user voice collected by the microphone.

[0015] That is, in response to the user turning on the first switch, the terminal device turns on the first voice control function. When the terminal device turns on the recording channel corresponding to the first voice control function, the first voice control function is turned on, so that the audio stream collected by the microphone is distributed to the first voice control function. After the first voice control function is turned on, the terminal device displays a first prompt icon to prompt the user that the first voice control function is turned on.

[0016] In combination with the first aspect, in a possible implementation, in response to the user's operation of turning on the second voice control function, the terminal device turns on the recording channel corresponding to the second voice control function and turns off the recording channel corresponding to the first voice control function.

[0017] In this method, the second voice control function is enabled and the first voice control function is disabled by switching the recording channel. After the recording channel corresponding to the second voice control function is enabled and the recording channel corresponding to the first voice control function is disabled, the audio stream collected by the microphone will be distributed to the second voice control function instead of the first voice control function.

[0018] In combination with the first aspect, in a possible implementation, after executing the second instruction corresponding to the second voice, the terminal device closes the recording channel corresponding to the second voice control function and opens the recording channel corresponding to the first voice control function.

[0019] In this method, after the terminal device is awakened and a voice command is executed, the second voice control function is automatically disabled. After the second voice control function is disabled, the first voice control function is automatically enabled. The audio stream collected by the microphone is distributed to the first voice control function, but not to the second voice control function.

[0020] In conjunction with the first aspect, in one possible implementation, after the terminal device stops displaying the second prompt icon and displays the first prompt icon, in response to a user's operation to activate the first function, the terminal device activates the first function; in response to the activation of the first function, the terminal device stops displaying the first prompt icon and displays the second prompt icon. After the first function is activated, the terminal device collects user voice via a microphone. In response to the user's operation to deactivate the first function, the terminal device stops displaying the second prompt icon and displays the first prompt icon.

[0021] Among them, the terminal device turning on the first function includes: the terminal device turning on the recording channel corresponding to the first function.

[0022] In combination with the first aspect, in a possible implementation, in response to the first function being activated, the terminal device closes the recording channel corresponding to the first voice control function.

[0023] In this method, when other applications (for example, the first function) collect user voice through the microphone, the visible and speak function (the first voice control function) is suspended (exit occupying the microphone), which can avoid false triggering of voice control and affecting the user's normal use of the terminal device.

[0024] In combination with the first aspect, in a possible implementation, in response to the user turning off the first switch, the terminal device turns off the recording channel corresponding to the first voice control function.

[0025] In combination with the first aspect, in a possible implementation, the operation of enabling the second voice control function includes: long pressing the power button on the terminal device, or voice inputting a wake-up word.

[0026] In conjunction with the first aspect, in one possible implementation, while the terminal device displays the first prompt icon, the terminal device receives a third voice captured by a microphone; the third voice includes the terminal device's wake-up word; the terminal device continues to display the first prompt icon, and the terminal device obtains the user's voice through the recording channel corresponding to the first voice control function.

[0027] In this method, the terminal device displays a first prompt icon indicating that the first voice control function has been turned on. The terminal device receives the wake-up word input by the user's voice, and the first voice control function is still available. The display interface of the terminal device does not change, and it is still the prompt icon of the first voice control function, and the recording channel is not switched.

[0028] In a second aspect, a method for handling concurrent conflicts in recorded voice is provided, the method comprising: a terminal device displays a first interface, the first interface comprising a first switch; in response to a user operation of turning on the first switch, the terminal device turns on a recording channel corresponding to a voice control function; the recording channel is used to obtain the user voice captured by the microphone; upon receiving the first voice captured by the microphone, the terminal device executes a first instruction corresponding to the first voice; in response to the user's first operation, the terminal device establishes a recording channel corresponding to a first application, and closes the recording channel corresponding to the voice control function.

[0029] In this method, when the first voice control function has started, if the first application occupies the microphone and collects the user's voice through the microphone, the first voice control function (visible and speak function) is suspended (exits occupying the microphone), which can avoid false triggering of voice control and affecting the user's normal use of the terminal device.

[0030] In combination with the second aspect, in a possible implementation, in response to the user turning on the first switch, the terminal device displays a first prompt icon, where the first prompt icon indicates that the voice control function has been turned on.

[0031] In combination with the second aspect, in a possible implementation, in response to the user's first operation, the terminal device stops displaying the first prompt icon and displays a second prompt icon, where the second prompt icon indicates that the voice control function is temporarily unavailable.

[0032] In combination with the second aspect, in one possible implementation, in response to the user's second operation, the terminal device closes the recording channel corresponding to the first application; in response to the recording channel corresponding to the first application being closed, the terminal device opens the recording channel corresponding to the voice control function.

[0033] According to a third aspect, a method for handling concurrent conflicts in voice recording is provided, the method comprising: in response to a first operation of a user, a terminal device establishing a recording channel corresponding to a first application; the recording channel is used to obtain the user voice captured by a microphone; the terminal device displays a first interface, the first interface comprising a first switch; the first switch is a switch corresponding to a voice control function; an operation of turning on the first switch by the user is received; in response to the operation of turning on the first switch by the user, the terminal device displays a prompt icon indicating that the voice control function is temporarily unavailable.

[0034] In this method, when a user turns on the first switch to enable the voice control function (the "see and talk" function), if other applications (such as a recording application, a phone application, an audio and video call application, etc.) are currently collecting the user's voice through the microphone, the terminal device temporarily disables the "see and talk" function, and the "see and talk" function is temporarily unavailable. This can prevent the voice control function from being triggered by mistake and affecting the user's normal use of the terminal device.

[0035] In combination with the third aspect, in one possible implementation, in response to the user's second operation, the terminal device closes the recording channel corresponding to the first application; in response to the recording channel corresponding to the first application being closed, the terminal device opens the recording channel corresponding to the voice control function.

[0036] In this method, when the first application exits and occupies the microphone, the voice control function (see-and-speak function) is automatically turned on.

[0037] In combination with the third aspect, in a possible implementation, after the terminal device turns on the recording channel corresponding to the voice control function, when the terminal device receives the first voice collected by the microphone, the terminal device executes the first instruction corresponding to the first voice.

[0038] In a fourth aspect, a terminal device is provided, comprising: a processor and a memory; the memory is used to store computer-executable instructions, and when the terminal device is running, the processor executes the computer-executable instructions stored in the memory to enable the terminal device to execute a method as described in any one of the first, second or third aspects above.

[0039] In a fifth aspect, a computer-readable storage medium is provided, which stores instructions. When the computer-readable storage medium is run on a computer, the computer can execute any one of the methods in the first, second or third aspects above.

[0040] Among them, the technical effects brought about by any design method in the fourth and fifth aspects can refer to the technical effects brought about by different design methods in the first, second or third aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1A is a schematic diagram of a scenario example of a method for controlling a terminal device by voice;

[0042] FIG1B is a schematic diagram of another scenario example of a method for controlling a terminal device by voice;

[0043] FIG2 is a schematic diagram of an example scenario for setting a preset switch for a voice control function according to an embodiment of the present application;

[0044] FIG3A is a first schematic diagram of an example of a voice control scenario applicable to the method for handling concurrent conflicts in recorded voices provided in an embodiment of the present application;

[0045] FIG3B is a second schematic diagram of an example of a voice control scenario applicable to the method for handling concurrent conflicts in recorded voices provided in an embodiment of the present application;

[0046] FIG3C is a third schematic diagram of an example of a voice control scenario applicable to the method for handling concurrent conflicts in recorded voices provided in an embodiment of the present application;

[0047] FIG4A is a schematic diagram of a software architecture of a terminal device provided in an embodiment of the present application;

[0048] FIG4B is a schematic diagram of the interaction flow of various modules during the activation and use of the voice assistant function;

[0049] FIG4C is a schematic diagram of the interaction flow of various modules during the activation and use of the see-and-speak function;

[0050] FIG4D is a schematic diagram of the interaction flow of various modules during the activation and use of the recording function;

[0051] FIG5 is a schematic diagram of a scenario example 1 of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0052] FIG6 is a second schematic diagram of a scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0053] FIG7 is a third schematic diagram of a scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0054] FIG8 is a fourth schematic diagram of a scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0055] FIG9 is a fifth schematic diagram of a scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0056] FIG10 is a sixth schematic diagram of a scenario example of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0057] FIG11 is a flowchart of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0058] FIG12 is a second flow chart of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0059] FIG13 is a third flow chart of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0060] FIG14 is a fourth flow chart of a method for handling concurrent conflicts in recorded speech provided in an embodiment of the present application;

[0061] FIG15 is a schematic diagram of the interaction flow of various modules when the voice assistant function and the see-and-speak function are used concurrently;

[0062] FIG16 is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0063] FIG17 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0064] FIG18 is a schematic diagram of the structural composition of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0066] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the situation where A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0067] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0068] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0069] Voice control function:

[0070] Many devices support voice control. These devices use microphones to capture user voice, analyze and recognize it, and execute commands corresponding to the user's voice, enabling users to control the device through voice.

[0071] Generally speaking, in order to save power consumption of terminal devices and avoid false triggering, the voice control function needs to be enabled before it can be used. For example, the user inputs a preset word (called a wake-up word) into the terminal device through voice to wake up the terminal device. After the terminal device is awakened, it can execute the command corresponding to the user's voice, which means that the voice control function is enabled. For example, the user can turn on or off the voice control function by turning on or off a preset switch in the human-computer interaction interface of the terminal device.

[0072] The voice control function may have different names on different terminal devices, such as "Voice Control," "Smart Voice," "Voice Assistant," "See and Speak," "Voice Command," "Free Command," and "Smart AI." The specific implementation of voice control functions with different names may also differ.

[0073] The following examples illustrate several different implementations of the voice control function.

[0074] Voice Assistant:

[0075] Before the user uses the voice assistant to control the terminal device, the terminal device needs to be woken up first. In one example, the wake-up word input by the user to the terminal device is detected, and the terminal device is awakened. In another example, the terminal device is awakened by detecting the user's long press of the power button. In yet another example, the terminal device is awakened by detecting the breath generated when the user inputs voice to the terminal device. Generally speaking, before the terminal device is awakened, the microphone of the terminal device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone is only detected for wake-up words at the kernel layer, and the corresponding recording channel of the voice assistant is not started in the system and driver of the terminal device.

[0076] In response to a user action (e.g., receiving a wake-up word via user voice input), the terminal device wakes up, and the system and driver activate the corresponding recording channel for the voice assistant. After the terminal device wakes up, the voice (audio stream) collected by the microphone is sent to the voice assistant application for processing via the corresponding recording channel of the voice assistant. In this way, the terminal device can execute the instructions corresponding to the user's voice, enabling the user to control the terminal device through voice; it can also realize functions such as dialogue with the user.

[0077] Illustratively, FIG1A shows a schematic diagram of an example scenario in which a user uses a voice assistant to control a mobile phone. As shown in FIG1A , the mobile phone displays a desktop interface, and the user inputs the voice "Hello YOYO" into the mobile phone. In response to receiving the wake-up word "Hello YOYO", the mobile phone is awakened. Illustratively, after the mobile phone is awakened, the voice "I'm here" is played to prompt the user that the mobile phone has been awakened. In one example, after the mobile phone is awakened, a voice assistant prompt card is displayed, indicating that the mobile phone has been awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. Illustratively, as shown in FIG1A , the user inputs the voice "open video" into the mobile phone, the mobile phone parses and recognizes the user's voice, and executes the instruction corresponding to the voice "open video". Illustratively, in response to receiving the user's voice "open video", the mobile phone starts the video application.

[0078] For example, Figure 1B shows a schematic diagram of another scenario example in which a user uses a voice assistant to control a mobile phone. As shown in Figure 1B, the mobile phone displays the desktop interface, and the user long-presses the power button. In response to receiving the user's long-press of the power button operation, the mobile phone is awakened. In one example, after the mobile phone is awakened, a voice assistant prompt card is displayed, indicating that the mobile phone has been awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. For example, as shown in Figure 1B, the user inputs the voice "open video" into the mobile phone, the mobile phone parses and recognizes the user's voice, and executes the instruction corresponding to the voice "open video". For example, in response to receiving the user's voice "open video", the mobile phone starts the video application.

[0079] In some implementations, after the terminal device is awakened, the user can give a command to the terminal device by inputting voice into the terminal device, and the terminal device executes the command corresponding to the voice. After the terminal device executes a command, or if the terminal device does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after being awakened, the terminal device will no longer respond to the command given by the user through voice. For example, the terminal device will turn off the recording channel corresponding to the voice assistant. The user needs to input the wake-up word into the terminal device again to wake up the terminal device before he can give commands to the terminal device by inputting voice into the terminal device again. In other words, after the terminal device is awakened, it enters a "short reception" state and can respond to the command given by the user through voice within a short period of time (for example, within 8 seconds).

[0080] In some implementations, when a terminal device is connected to a network, it can wake up and enter a continuous conversation scenario, allowing the user to continue the conversation with the terminal device. After each announcement, the terminal device will continue to pick up the audio without having to wake up again until the user exits the continuous conversation using a command such as "Exit".

[0081] You can see that:

[0082] It can be seen that it is implemented locally by the terminal device and does not require a network connection.

[0083] Visible and Talkable is controlled by a preset switch. Users can enable Visible and Talkable by turning on the preset switch, and can exit Visible and Talkable by turning off the preset switch.

[0084] Taking a mobile phone as an example, as shown in FIG2 , a user can open the settings function of mobile phone 100 by, for example, clicking the "Settings" application icon on the desktop. In response to the user clicking the "Settings" application icon, mobile phone 100 displays "Settings" interface 101. "Settings" interface 101 includes a "Smart Voice" option 102, which is used to set up the Smart Voice function. In response to the user clicking "Smart Voice" option 102, mobile phone 100 displays "Smart Voice" interface 103. "Smart Voice" interface 103 includes a "See and Speak" option 104. The user can click "See and Speak" option 104 to set options related to the See and Speak function. In response to the user clicking "See and Speak" option 104, mobile phone 100 displays "See and Speak" interface 105. Optionally, "See and Speak" interface 105 includes a prompt 106 that prompts the user on how to use the See and Speak function. "See and Speak" interface 105 also includes a "See and Speak" switch 107. The user can click "See and Speak" switch 107 to turn the "See and Speak" switch on or off. In one example, in response to receiving a user click on "See and Speak" switch 107, the "See and Speak" switch of mobile phone 100 is turned on, enabling the "See and Speak" feature. Optionally, "See and Speak" interface 105 displays a prompt message 108 to inform the user that the "See and Speak" feature has been successfully enabled.

[0085] In one embodiment, after the "See and Talk" function is turned on, the mobile phone 100 displays a first prompt icon, which indicates that the "See and Talk" function is turned on. For example, as shown in Figure 2, after the "See and Talk" switch 107 is turned on, the status bar of the mobile phone 100 display interface displays a prompt icon 10a, indicating that the "See and Talk" function is turned on.

[0086] In one scenario, the preset switch for Visible and Talk on the terminal device is not turned on, and the microphone of the terminal device is not turned on. When the preset switch for Visible and Talk is turned on, the terminal device starts the microphone and starts the corresponding recording channel for Visible and Talk in the system and driver. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talk application for processing through the corresponding recording channel of Visible and Talk, allowing users to control the terminal device through voice.

[0087] In another scenario, the preset switch corresponding to the Visible and Talk function of the terminal device is not turned on, and the microphone of the terminal device operates in a power-saving mode (for example, searching for signals at a lower power) to pick up sounds from the surrounding environment. When the preset switch corresponding to the Visible and Talk function is turned on, the system and driver of the terminal device start the corresponding recording channel of the Visible and Talk function. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talk application for processing through the corresponding recording channel of the Visible and Talk function, allowing users to control the terminal device through voice.

[0088] When Visible and Talking is enabled, the corresponding recording channel is activated in the terminal device's system and driver. The terminal device enters a "long reception" state, continuously collecting surrounding sounds. Users can give commands to the terminal device at any time through voice, without having to enter a wake-up word to wake the terminal.

[0089] In one implementation, after any function on the terminal device turns on the voice recording function of the terminal device (turns on the microphone and establishes a recording channel), the terminal device will send a prompt message to the user to remind the user that the terminal device is in a voice collection state. In this way, the user's privacy can be avoided from being leaked. For example, after the visible and audible function is turned on, the terminal device enters a state of continuous voice collection, and a third prompt icon is displayed on the display interface of the terminal device. The third prompt icon indicates that the recording channel is turned on, which is used to remind the user that the microphone is collecting voice. For example, as shown in Figure 2, the status bar of the display interface of the mobile phone 100 displays a prompt icon 10b, indicating that the recording channel is turned on, which is used to remind the user that the microphone is collecting voice.

[0090] When the "See and Speak" feature is enabled, the device establishes a recording channel and continuously collects the user's voice through the microphone. The user can input voice into the device at any time. The device analyzes and recognizes the user's voice input and executes the corresponding command.

[0091] 3A , the mobile phone 100 displays a desktop interface 120. The user inputs a voice message “open video” to the mobile phone 100. In response to receiving the user input voice message “open video”, the mobile phone 100 starts a video application and displays a user interface 121 of the video application.

[0092] 3B , the mobile phone 100 displays the desktop interface 120 and receives a user voice input of “swipe left.” In response to receiving the user voice input of “swipe left,” the mobile phone 100 displays the negative one screen interface 122 .

[0093] In another example, as shown in FIG3C , mobile phone 100 displays interface 123 of a short video application. A user inputs a voice message "return to desktop" into mobile phone 100. In response to receiving the user's voice message "return to desktop," mobile phone 100 executes the command corresponding to the voice message "return to desktop" and displays desktop interface 120.

[0094] It should be noted that the “collecting user voice through a microphone” and “occupying the microphone” described in the embodiments of the present application refer to the process of starting a recording channel in the system and driver of the terminal device and processing the audio stream of the user voice collected by the microphone. In other scenarios, the terminal device turns on the microphone to pick up the surrounding environment, but does not establish a recording channel in the system and driver, which does not fall under the circumstances of “collecting user voice through a microphone” and “occupying the microphone” described in the embodiments of the present application. For example, before the terminal device is awakened, the microphone of the terminal device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone is only detected for wake-up words at the kernel layer, and the corresponding recording channel of the voice assistant is not started in the system and driver of the terminal device. This situation does not fall under the circumstances of “collecting user voice through a microphone” and “occupying the microphone” described in the embodiments of the present application.

[0095] The following describes, in conjunction with the software architecture of the terminal device, the process of collecting user voice through the microphone by various applications and functions in the terminal device.

[0096] The method for handling concurrent conflicts in recorded voice provided in the embodiment of the present application can be applied to a terminal device 100 including a microphone. The terminal device 100 may include a mobile phone, a tablet computer, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (e.g., a smart TV, a smart screen, a large screen, a smart speaker, a smart air conditioner, etc.), a personal digital assistant (PDA), a wearable device (e.g., a smart watch, a smart bracelet, etc.), an in-vehicle device, a virtual reality device, etc., and the embodiment of the present application does not impose any restrictions on this.

[0097] In the embodiment of the present application, the terminal device 100 is a terminal device that can run an operating system and install applications. Optionally, the operating system running on the terminal device can be system, system, System, etc.

[0098] In some embodiments, the software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture or a cloud architecture. Taking the system as an example, the software structure of the terminal device 100 is exemplarily described.

[0099] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0100] As shown in Figure 4A, the application layer may include a series of application packages APK. For example, applications such as camera, recording, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video call, short message, etc. The application layer also includes a voice processing engine for parsing, recognizing and processing voice. In an embodiment of the present application, the application layer includes a voice control APK, which is used to provide the voice control function of the terminal device 100 (for example, the see-and-speak function and the voice assistant function). Exemplarily, the voice control APK includes a see-and-speak APK and a voice assistant application; the see-and-speak APK is used to provide the see-and-speak function of the terminal device 100, and the voice assistant application is used to provide the voice assistant function of the terminal device 100; wherein, the see-and-speak APK includes a service presentation module, a service processing module, a voice processing module, etc.

[0101] Among them, the business presentation module is used to provide information displayed to the user on the user interface. For example, it provides prompt information displayed on the display interface of the terminal device 100. The business processing module is used to control the relevant processes of recording voice of the Visible and Speakable function. For example, it controls the opening and closing of the Visible and Speakable function, controls whether the Visible and Speakable function is available or unavailable, etc. The business processing module may include a recording management module, a status management module, a monitoring module, etc. The monitoring module is used to monitor whether other applications occupy the microphone. The recording management module is used to control the opening and closing of the recording channel corresponding to the Visible and Speakable function; for example, according to whether the Visible and Speakable switch is turned on or off, the recording channel corresponding to the Visible and Speakable function is turned on or off. The status management module is used to manage the status of the Visible and Speakable function, such as turned on, turned off, temporarily unavailable, etc.

[0102] The speech processing module is responsible for recognizing and processing the collected user speech. For example, the speech processing module invokes the speech processing engine to perform text parsing, semantic understanding, and recognition on the audio stream collected by the microphone, and then converts the user's speech into voice commands. The speech processing module sends the voice commands to the multi-mode control module in the application framework layer, which then distributes the voice commands to the corresponding application for execution.

[0103] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0104] As shown in FIG4A , the application framework layer may include a window manager, a content provider, a view system, a resource manager, an activity manager service (AMS), an application manager (PMS), a multi-mode control module, an audio module, and the like.

[0105] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0106] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0107] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0108] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0109] The AMS is primarily responsible for starting, switching, and scheduling the four major components of the system, as well as managing and scheduling application processes. Its responsibilities are similar to those of the process management and scheduling module in an operating system. When a process or component is started, the request is passed to the AMS via the Binder communication mechanism, and the AMS then handles it uniformly.

[0110] PMS handles package management related tasks, such as application installation and uninstallation. It is also used to provide all the information of an application, such as startup and shutdown.

[0111] The multi-mode control module is used to manage the execution of voice commands, for example, sending voice commands to applications so that the applications execute the voice commands.

[0112] The audio module is used to handle audio-related functions, such as recording audio, playing audio, etc. It includes a recording module, which is used to manage the audio recording process.

[0113] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0114] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0115] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0116] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0117] A 2D graphics engine is a drawing engine for 2D drawings.

[0118] The Android runtime is responsible for scheduling and management of the Android system. The Android runtime includes the core library and the virtual machine.

[0119] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0120] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0121] The kernel layer is the layer between hardware and software. It can include display drivers, sensor drivers, audio drivers, Wi-Fi drivers, and more. It also includes a wake-up word detection module, which detects whether a wake-up word has been received based on the audio captured by the microphone. It also includes an audio digital signal processor (ADSP), which distributes and processes the audio captured by the microphone, such as establishing an application recording channel and distributing audio streams to the application.

[0122] In combination with the software architecture shown in Figure 4A, Figure 4B shows the interaction process of each module during the activation and use of the voice assistant function.

[0123] Exemplarily, referring to Figure 4B, after the voice assistant switch is turned on, the terminal device will turn on the microphone, and the microphone will work in a power-saving mode (such as searching for signals at lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone will be transmitted to the wake-up word detection module for wake-up word detection. If the user inputs a wake-up word to the terminal device and the wake-up word detection module detects the wake-up word, it will notify the voice assistant application that the terminal device is awakened. The voice assistant application starts the voice assistant function and requests the audio module to establish a recording channel corresponding to the voice assistant function. The audio module requests the ADSP to establish a recording channel. In this way, a recording channel corresponding to the voice assistant function is established. The audio stream collected by the microphone will be distributed to the voice assistant application for processing through the recording channel corresponding to the voice assistant function.

[0124] Optionally, after a preset time (e.g., 8 seconds), the recording channel corresponding to the voice assistant function is turned off.

[0125] In conjunction with the software architecture shown in FIG4A , FIG4C shows the interaction flow of various modules during the activation and use of the see-is-say function.

[0126] For example, referring to Figure 4C , the terminal device receives a user's request to turn on the "See and Speak" switch, enabling the "See and Speak" feature. The "See and Speak" application requests the audio module to establish a recording channel corresponding to the "See and Speak" feature. The audio module requests the ADSP to establish a recording channel. This establishes a recording channel corresponding to the "See and Speak" feature. The audio stream subsequently captured by the microphone is distributed to the "See and Speak" application for processing via the "See and Speak" recording channel.

[0127] Similarly, other applications or functions on the terminal device can also request to establish corresponding recording channels. For example, referring to Figure 4D, the terminal device receives the user's operation to start recording and starts the recording function. The recording application requests the audio module to establish a recording channel corresponding to the recording function. The audio module requests the ADSP to establish a recording channel. In this way, a recording channel corresponding to the recording function is established. The audio stream collected by the microphone will be distributed to the recording application for processing through the recording channel corresponding to the recording function.

[0128] After the voice control function is turned on, the terminal device establishes a recording channel in the system and driver, and the processing module for the voice control function obtains the audio stream collected by the microphone. In order to obtain the voice input by the user, parse and recognize the user's voice, execute the instructions corresponding to the user's voice, and enable the user to control the terminal device through voice. For example, after the "See and Speak" function is turned on, the terminal device starts the corresponding recording channel of "See and Speak", and the microphone continues to collect the user's voice. The user can control the terminal device through voice. For example, after the voice assistant is turned on (the terminal device is awakened), the terminal device starts the corresponding recording channel of the voice assistant. Within a preset period of time (for example, within 8 seconds), the user can give a command to the terminal device by inputting voice into the terminal device, and the terminal device executes the instruction corresponding to the voice.

[0129] There are also some applications or functions on the terminal device, such as recording applications, telephone applications, audio and video call applications, etc., which will also establish recording channels and collect user voice through the microphone.

[0130] If the See-and-Talk feature is enabled and a user uses an application (or function) on a terminal device that has voice recording capabilities, such as a recording app, phone app, audio and video call app, or voice assistant, a concurrent conflict may occur, causing abnormal terminal device processes. For example, a user uses the phone app to make a call, and the terminal device collects the user's voice. If the voice control function executes a corresponding command in response to the user's voice, this will cause an erroneous operation. For example, with the See-and-Talk feature enabled, a user uses mobile phone 100 to answer a call, and the microphone of mobile phone 100 collects the user's voice. During the call, the user says, "The cat has crawled back onto the table." The voice control function of mobile phone 100 mistakenly executes the voice command corresponding to "Return to the table," and the phone app is pushed back to the background. For example, with the See-and-Talk feature enabled, a user uses mobile phone 100 to make a call, and the microphone of mobile phone 100 collects the user's voice. During the call, the user says, "The strong wind broke the branch." The voice control function of mobile phone 100 mistakenly executes the voice command corresponding to "Hang up," and the current call is hung up.

[0131] An embodiment of the present application provides a method for handling concurrent conflicts in voice recording. When other applications collect user voice through the microphone, the visible and speak function is suspended (exit occupying the microphone) to avoid false triggering of voice control and affecting the user's normal use of the terminal device.

[0132] Establishing a Recording Channel In some embodiments, when the See-and-Talk feature is enabled and the See-and-Talk feature is collecting user voice through the microphone, other applications (e.g., a recording application, a phone application, an audio and video call application, etc.) begin collecting user voice through the microphone. In response to other applications (e.g., a recording application, a phone application, an audio and video call application, etc.) starting to collect user voice through the microphone, the See-and-Talk feature is temporarily unavailable, and the See-and-Talk feature temporarily stops occupying the microphone; the terminal device prompts the user that the See-and-Talk feature is temporarily unavailable. In one implementation, when the See-and-Talk feature switches from enabled to temporarily unavailable, the status bar of the terminal device display interface no longer displays the first prompt icon, and switches to displaying a second prompt icon, which indicates that the See-and-Talk feature is temporarily unavailable.

[0133] In one example, as shown in FIG5(a), the mobile phone 100 has enabled the "See and Speak" feature. For example, the user uses the method shown in FIG2 to turn on the "See and Speak" switch on the mobile phone 100, thereby enabling the "See and Speak" feature. The mobile phone 100 exits the settings application and displays the desktop interface 120. The status bar of the display interface of the mobile phone 100 displays a prompt icon 10a, indicating that the "See and Speak" feature is enabled. Optionally, the "See and Speak" feature has enabled the recording channel, and the status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b to remind the user that the microphone is collecting voice.

[0134] Mobile phone 100 receives a video call request from another device. As shown in FIG5(b), mobile phone 100 displays user interface 124. For example, user interface 124 includes a prompt message "XX invites you to a video call," notifying the user that mobile phone 100 has received the video call request. User interface 124 also includes an "Answer" button 125, which the user can click to initiate the video call.

[0135] The mobile phone 100 detects the user clicking the “Answer” button 125 . In response to the user clicking the “Answer” button 125 , the mobile phone 100 starts the video call and displays the user interface 126 shown in FIG. 5( c ).

[0136] Mobile phone 100 initiates a video call and uses the microphone to collect user voice. The "See-and-Talk" feature is temporarily unavailable, and the "See-and-Talk" feature no longer occupies the microphone. At this point, the preset switch for "See-and-Talk" remains on. For example, as shown in FIG5(c), the status bar on the display interface of mobile phone 100 switches from displaying prompt icon 10a to displaying prompt icon 10c, indicating that the "See-and-Talk" feature is temporarily unavailable. Optionally, mobile phone 100 also displays prompt message 127, notifying the user that the microphone is occupied by another application and that the voice control (See-and-Talk) feature is temporarily unavailable. In one implementation, prompt message 127 remains on the display interface for a preset length of time (e.g., 2 seconds) and then disappears. The "See-and-Talk" feature temporarily stops occupying the microphone, but the video call application continues to collect user voice through the microphone. For example, the status bar on the display interface of mobile phone 100 still displays prompt icon 10b, notifying the user that the microphone is collecting voice.

[0137] In this example, while the See-and-Talk feature is capturing user voice through the microphone, the video call app preempts the microphone, temporarily suspending the See-and-Talk feature. This means that the See-and-Talk feature conflicts with the video call app's voice recording. While the video call app is capturing user voice through the microphone, the See-and-Talk feature is temporarily suspended.

[0138] In another example, as shown in FIG6 (a), the mobile phone 100 has enabled the "See and Speak" feature. For example, the user uses the method shown in FIG2 to turn on the "See and Speak" switch on the mobile phone 100, thereby enabling the "See and Speak" feature. The mobile phone 100 exits the settings application and displays the desktop interface 120. The status bar of the mobile phone 100 display interface displays a prompt icon 10a, indicating that the "See and Speak" feature is enabled. Optionally, the "See and Speak" feature activates the microphone, and the status bar of the mobile phone 100 display interface also displays a prompt icon 10b to remind the user that the microphone is collecting voice.

[0139] The desktop interface 120 includes an icon for a recording application called "Voice Recorder." The user can click the "Voice Recorder" icon to launch the recording application. Exemplarily, in response to receiving a click on the "Voice Recorder" icon, the mobile phone 100 launches the recording application. As shown in FIG6( b ), the mobile phone 100 displays a voice recorder interface 128. The voice recorder interface 128 includes a start button 1281. The user can click the start button 1281 to launch the recording function of the voice recorder.

[0140] Exemplarily, in response to receiving a user click operation on the start button 1281, the mobile phone 100 displays the recording interface 129. Optionally, the status bar of the mobile phone 100 display interface also displays a recording icon 10d, indicating that the recording application is recording voice. In other words, the recording application is collecting user voice through the microphone.

[0141] The recording application of the mobile phone 100 collects the user's voice through the microphone; in response to the recording application starting to collect the user's voice through the microphone, the See-and-Speak function is temporarily suspended, and the See-and-Speak function temporarily stops occupying the microphone. For example, as shown in FIG6(c), the status bar of the display interface of the mobile phone 100 displays a prompt icon 10c, which indicates that the See-and-Speak function is temporarily unavailable. Optionally, the recording interface 129 of the mobile phone 100 also displays a prompt message 127 to inform the user that the microphone is occupied by another application and the voice control (See-and-Speak) function is temporarily unavailable. In one implementation, the prompt message 127 remains on the display interface for a preset length of time (e.g., 2 seconds) and then disappears. The See-and-Speak function temporarily stops occupying the microphone, but the recording application still collects the user's voice through the microphone. For example, the status bar of the display interface of the mobile phone 100 still displays the prompt icon 10b to inform the user that the microphone is collecting voice.

[0142] In this example, while the See-and-Talk feature is capturing user voice through the microphone, the recording app preempts the microphone, temporarily suspending the See-and-Talk feature. This means that the See-and-Talk feature conflicts with the recording app's concurrent voice recording. While the recording app is capturing user voice through the microphone, the See-and-Talk feature is temporarily suspended.

[0143] In some embodiments, when a user turns on a preset switch to enable the "see and speak" feature, if other applications (e.g., a recording application, a phone application, an audio and video call application, etc.) are currently collecting the user's voice through a microphone, in response to the user turning on the preset switch, a switch on the terminal device display interface is turned on, temporarily disabling the "see and speak" feature, and the "see and speak" feature is temporarily unavailable. In one example, a second prompt icon is displayed on the terminal device display interface, indicating that the "see and speak" feature is temporarily unavailable.

[0144] For example, as shown in FIG7(a), a user is recording a voice using the recording application of the mobile phone 100. The mobile phone 100 displays a recording interface 129, and the recording application collects the user's voice through the microphone. For example, the status bar of the mobile phone 100 display interface displays a prompt icon 10b to prompt the user that the microphone is collecting voice. The status bar of the mobile phone 100 display interface also displays a recording icon 10d, indicating that the recording application is recording voice. In other words, the recording application is collecting the user's voice through the microphone.

[0145] In one example, while a recording application is recording voice through a microphone, a user activates a preset switch corresponding to "visible and speaking." For example, referring to FIG7(a), the mobile phone 100 receives a left swipe on the recording interface 129. In response to this swipe, the recording application exits the foreground and returns to the background. As shown in FIG7(b), the mobile phone 100 displays the desktop interface, and the status bar of the mobile phone 100 display shows a recording icon 10d, indicating that the recording application is recording voice. The user can access the settings function of the mobile phone 100; for example, the user clicks the "Settings" application icon on the desktop. As shown in FIG7(c), in response to the user clicking the "Settings" application icon, the mobile phone 100 displays the "Settings" interface 101. The "Settings" interface 101 includes a "Smart Voice" option 102, which is used to configure the smart voice function. Exemplarily, as shown in FIG7(d), in response to a user clicking on the "Smart Voice" option 102, the mobile phone 100 displays a "Smart Voice" interface 103. The "Smart Voice" interface 103 includes a "See and Speak" option 104. The user can click on the "See and Speak" option 104 to set options related to the "See and Speak" function. Exemplarily, referring to FIG7(e), in response to a user clicking on the "See and Speak" option 104, the mobile phone 100 displays a "See and Speak" interface 105. The "See and Speak" interface 105 includes a "See and Speak" switch 107. The user can click on the "See and Speak" switch 107 to turn the "See and Speak" function on or off.

[0146] In one example, referring to (f) of FIG7 , in response to receiving a user click operation on the “See and Speak” switch 107, the “See and Speak” switch 107 of the mobile phone 100 is turned on. At this time, the recording application is collecting the user's voice through the microphone, and the status bar of the mobile phone 100 display interface displays a recording icon 10d. The “See and Speak” function cannot be turned on temporarily, and the “See and Speak” function is temporarily unavailable. Optionally, the “See and Speak” interface 105 displays a prompt message 109 to prompt the user that the “See and Speak” function is temporarily unavailable. In one example, the status bar of the mobile phone 100 display interface displays a prompt icon 10c, indicating that the “See and Speak” function is temporarily unavailable.

[0147] In this scenario, after the "See and Speak" switch is turned on, the "See and Speak" function is temporarily unavailable. For example, referring to Figure 8, the mobile phone 100 displays the "See and Speak" interface 105, and the "See and Speak" switch 107 in the "See and Speak" interface 105 is turned on. The recording application is running in the background, and the status bar of the mobile phone 100 display interface displays the recording icon 10d, indicating that the recording application is recording voice. The status bar of the mobile phone 100 display interface also displays a prompt icon 10c, indicating that the "See and Speak" function is temporarily unavailable. Optionally, the status bar of the mobile phone 100 display interface also displays a prompt icon 10b to remind the user that the microphone is collecting voice.

[0148] In one example, a user performs a "swipe up" gesture at the bottom of the display screen of mobile phone 100. In response to receiving the user's "swipe up" gesture, mobile phone 100 exits the application interface of the "Settings" application and displays desktop interface 120. In this display interface, the status bar displays a recording icon 10d, indicating that the recording application is recording voice; the status bar also displays a prompt icon 10c, indicating that the see-to-speak function is temporarily unavailable. Optionally, the status bar of the display interface of mobile phone 100 also displays a prompt icon 10b to remind the user that the microphone is collecting voice.

[0149] In some embodiments, the "See and Talk" switch is turned on, and the "See and Talk" feature is temporarily unavailable because other applications (e.g., recording applications, phone applications, audio and video call applications, etc.) are using the microphone to collect user voice. When other applications that are occupying the microphone no longer use the microphone, the "See and Talk" feature is automatically enabled; the "See and Talk" feature collects user voice through the microphone. In one implementation, when the "See and Talk" feature is automatically enabled, the terminal device no longer displays the second prompt icon, and instead displays the first prompt icon.

[0150] In one example, as shown in FIG9(a), a user is making a video call using mobile phone 100, and mobile phone 100 displays user interface 126. The video call application uses a microphone to collect the user's voice, and the "see and talk" feature is temporarily unavailable. A prompt icon 10c is displayed in the status bar of the mobile phone 100 display interface, indicating that the "see and talk" feature is temporarily unavailable. The status bar of the mobile phone 100 display interface also displays a prompt icon 10b to remind the user that the microphone is collecting voice.

[0151] The Visible and Talk feature is temporarily unavailable. This means that the Visible and Talk switch is turned on, but the microphone is occupied by another application. When the microphone is no longer occupied by other applications, the terminal device automatically turns on the Visible and Talk feature.

[0152] Exemplarily, the user interface 126 includes a "hang up" button, and the user can click the "hang up" button to stop the video call. Exemplarily, as shown in FIG9( b ), in response to the user clicking the "hang up" button, the mobile phone 100 closes the video call application and displays the desktop interface 120.

[0153] After the video call application is closed, the microphone is no longer occupied; the "See and Talk" feature is successfully enabled. For example, as shown in FIG9(b), the status bar of the mobile phone 100 display interface no longer displays prompt icon 10c; instead, the status bar displays prompt icon 10a, indicating that the "See and Talk" feature is enabled. The "See and Talk" feature collects user voice through the microphone, and the status bar of the mobile phone 100 display interface also displays prompt icon 10b to remind the user that the microphone is collecting voice. Optionally, the mobile phone 100 also displays prompt message 130 to remind the user that the "See and Talk" feature is enabled.

[0154] In this example, the "See and Talk" feature is temporarily unavailable while the video call app is occupying the microphone. Once the video call app is no longer occupying the microphone, the "See and Talk" feature automatically turns on.

[0155] In another example, as shown in FIG10( a ), a user uses a mobile phone 100 to record a voice, and the mobile phone 100 displays a recording interface 129 . The recording application records voice through the microphone, and the visible-to-speak function is temporarily unavailable. For example, the status bar of the mobile phone 100 display interface displays a recording icon 10d, indicating that the recording application is recording voice. The status bar of the mobile phone 100 display interface also displays a prompt icon 10c, indicating that the visible-to-speak function is temporarily unavailable. Optionally, the status bar of the mobile phone 100 display interface also displays a prompt icon 10b to remind the user that the microphone is collecting voice.

[0156] The Visible and Talk feature is temporarily unavailable. This means that the Visible and Talk switch is turned on, but the microphone is occupied by another application. When the microphone is no longer occupied by other applications, the terminal device automatically turns on the Visible and Talk feature.

[0157] Exemplarily, the recording interface 129 includes a "Stop" button, and the user can click the "Stop" button to stop recording voice. Exemplarily, as shown in FIG10(b), in response to the user clicking the "Stop" button, the recording application stops recording voice, and the mobile phone 100 displays the recorder interface 128.

[0158] After the recording application stops recording voice through the microphone, it no longer occupies the microphone; the "visible and speak" function is successfully turned on. For example, as shown in (b) of Figure 10, the recording application no longer occupies the microphone, and the status bar of the mobile phone 100 display interface no longer displays the recording icon 10d; the "visible and speak" function is turned on, and the status bar of the mobile phone 100 display interface no longer displays the prompt icon 10c; the status bar displays the prompt icon 10a, indicating that the "visible and speak" function is turned on. The "visible and speak" function collects user voice through the microphone, and the status bar of the mobile phone 100 display interface also displays the prompt icon 10b to remind the user that the microphone is collecting voice. Optionally, the mobile phone 100 also displays a prompt message 130 to remind the user that the "visible and speak" function is turned on.

[0159] In this example, the "See and Talk" feature is temporarily unavailable while the recording app is occupying the microphone. Once the recording app is no longer occupying the microphone, the "See and Talk" feature automatically turns on.

[0160] On the terminal device, the voice control function needs to collect the user's voice through the microphone, so that the corresponding instructions can be executed according to the user's voice; if there are other applications or functions (such as recording applications, telephone applications, audio and video call applications, etc.), they also need to record voice through the microphone; the voice control function and other applications recording voice will cause concurrency conflicts. The method for handling concurrency conflicts of recorded voice provided in the embodiment of the present application is that when the switch of the voice control function is turned on, if there are other applications (or functions) occupying the microphone to record voice, the voice control function will be temporarily unavailable. In this way, it is avoided that the voice control function mistakenly executes the instructions corresponding to the voice, causing erroneous operations.

[0161] In the method for handling concurrent conflicts of recorded voice provided in an embodiment of the present application, when the preset switch corresponding to the voice control function is not turned on, the voice control function is turned off. When the preset switch corresponding to the voice control function is turned on, if the voice control function is temporarily unavailable, the status bar of the terminal device display interface displays a second prompt icon (for example, prompt icon 10c). When the preset switch corresponding to the voice control function is turned on, if the voice control function is successfully turned on, the status bar of the terminal device display interface displays a first prompt icon (for example, prompt icon 10a). In this way, the user can easily know whether the voice control function is successfully turned on through the prompt icon.

[0162] With reference to the example scenarios shown in Figures 5 and 6 of the audio driver, in some embodiments, when the See-and-Talk feature is enabled and the user's voice is being collected through the microphone, another application (e.g., a recording application, a phone application, an audio or video call application, etc.) requests to collect the user's voice through the microphone. In response to the other application (e.g., a recording application, a phone application, an audio or video call application, etc.) starting to collect the user's voice through the microphone, the See-and-Talk feature temporarily stops occupying the microphone.

[0163] In combination with the software architecture of the terminal device 100 shown in FIG4A , FIG11 shows a flow chart of a method for handling concurrent conflicts in recorded voice provided by an embodiment of the present application in this scenario.

[0164] As shown in FIG11 , the method for handling concurrent voice recording conflicts provided in the embodiment of the present application may include:

[0165] S1101: Upon receiving an operation from a user to turn on a "see and speak" function switch, the recording management module queries the recording module for microphone occupancy status.

[0166] Referring to Figures 4B-4D , each application requests the ADSP to establish a recording channel through the recording module. The recording module manages the establishment status of recording channels. If a recording channel corresponding to an application exists, it indicates that the microphone is occupied.

[0167] The recording management module queries the recording module for the microphone occupancy status, which includes occupied and unoccupied.

[0168] S1102: The recording module returns to the recording management module that the microphone is not occupied.

[0169] S1103: In response to the user turning on the "see and speak" function switch, the recording management module turns on the "see and speak" function.

[0170] In one example, a user can enable the "See and Speak" feature by turning on the "See and Speak" switch. For example, the user turns on the "See and Speak" switch on mobile phone 100 using the method shown in FIG2 . The recording management module determines that the microphone occupancy status is unoccupied and, in response to the user turning on the "See and Speak" switch, enables the "See and Speak" feature.

[0171] S1104. The recording management module notifies the recording module to establish a recording channel corresponding to the see-and-speak function.

[0172] In one implementation, the recording management module notifies the recording module to establish a recording channel (AudioRecord) corresponding to the see-and-speak function.

[0173] S1105: The recording module notifies the ADSP to establish a recording channel.

[0174] The recording module notifies the ADSP to turn on the microphone and collect the audio stream. The ADSP establishes a recording channel (AudioRecord) corresponding to the See-and-Talk feature, and the recording module distributes the audio stream collected by the microphone to the See-and-Talk application.

[0175] S1106. The recording management module notifies the status management module to record that the status of the see-and-speak function is enabled.

[0176] For example, the status of the see-and-tell function may include enabled, disabled, temporarily unavailable, etc.

[0177] S1107: The recording management module also notifies the monitoring module to monitor the microphone occupancy status.

[0178] S1108. The monitoring module registers monitoring with the recording module.

[0179] In one implementation, if an application needs to obtain an audio stream, it needs to request the recording module to establish an AudioRecord corresponding to the application. The recording module can determine the microphone occupancy status based on the activated AudioRecord; that is, the recording module can monitor the microphone occupancy status.

[0180] S1109 : Other applications request to record voice through the microphone.

[0181] For example, referring to the scenario shown in Figure 5, the video call application requests to record voice through the microphone. In one implementation, the video call application requests the recording module to create an AudioRecord corresponding to the video call application.

[0182] For example, referring to the scenario shown in Figure 6, a recording application requests to record voice through a microphone. In one implementation, the recording application requests the recording module to create an AudioRecord corresponding to the recording application.

[0183] S1110 : The recording module notifies the monitoring module that the microphone is occupied by other applications.

[0184] S1111. The monitoring module notifies the recording management module that the microphone is occupied by other applications.

[0185] S1112. The recording management module notifies the recording module to close the recording channel corresponding to the see-and-speak function.

[0186] When the recording management module receives a notification that the microphone is occupied by another application, it temporarily stops collecting audio streams through the microphone. In one implementation, the recording management module notifies the recording module to disable the AudioRecord corresponding to the See-and-Talk feature. This prevents the audio stream collected by the microphone from being distributed to the See-and-Talk application.

[0187] S1113: The recording module notifies the ADSP to close the previous recording channel and establish a new recording channel.

[0188] ADSP establishes a recording channel corresponding to other applications, and the audio stream collected by the microphone will be distributed to the application.

[0189] S1114. The recording management module notifies the status management module to update the status of the see-and-speak function to temporarily unavailable, and notifies the service presentation module to issue a prompt on the user interface that the see-and-speak function is temporarily unavailable.

[0190] S1115. The service presentation module issues a prompt message on the user interface indicating that the see-and-speak function is temporarily unavailable, and displays a prompt icon indicating that the see-and-speak function is temporarily unavailable.

[0191] In one implementation, the service presentation module issues a prompt on the user interface to inform the user that the see-and-speak feature is temporarily unavailable. For example, as shown in FIG5(c) and FIG6(c), the status bar of the mobile phone 100 display interface displays a prompt icon 10c, indicating that the see-and-speak feature is temporarily unavailable. The mobile phone 100 also displays a prompt message 127 to inform the user that the microphone is occupied by another application and that the see-and-speak feature is temporarily unavailable.

[0192] Referring to the scenario example shown in Figure 7, in some embodiments, when the user turns on the preset switch to enable the See-and-Speak function, there are other applications (such as recording applications, telephone applications, audio and video call applications, etc.) that are collecting user voice through the microphone, and the terminal device temporarily does not enable the See-and-Speak function, and the See-and-Speak function is temporarily unavailable.

[0193] In combination with the software architecture of the terminal device 100 shown in FIG4A , FIG12 shows a flow chart of a method for handling concurrent conflicts in recorded voice provided by an embodiment of the present application in this scenario.

[0194] As shown in FIG12 , the method for handling concurrent voice recording conflicts provided in the embodiment of the present application may include:

[0195] S1201: Upon receiving an operation from the user to turn on the "see and speak" function switch, the recording management module queries the recording module for microphone occupancy status.

[0196] S1202: The recording module returns information that the microphone is occupied to the recording management module.

[0197] In one implementation, when each application requests to capture an audio stream through the microphone, it requests the recording module to establish an AudioRecord corresponding to the application. The recording module can determine the microphone occupancy status based on the activated AudioRecord; that is, the recording module can monitor the microphone occupancy status.

[0198] S1203: The recording management module notifies the status management module to update the status of the visible-to-speak function to temporarily unavailable.

[0199] When the user turns on the "See Now Talk" feature, the Recording Management Module queries the Recording Module for microphone occupancy status. If the microphone is occupied, indicating that another application is using the microphone to capture audio streams, the "See Now Talk" feature is temporarily disabled. The Recording Management Module notifies the Status Management Module to update the "See Now Talk" feature to "temporarily unavailable."

[0200] S1204: The recording management module notifies the service presentation module to issue a prompt on the user interface that the see-and-speak function is temporarily unavailable.

[0201] S1205. The service presentation module issues a prompt message on the user interface indicating that the visible-to-speak function is temporarily unavailable, and displays a prompt icon indicating that the visible-to-speak function is temporarily unavailable.

[0202] In one implementation, the service presentation module issues a prompt on the user interface to inform the user that the see-and-speak feature is temporarily unavailable. For example, as shown in FIG7(f), the status bar of the mobile phone 100 display interface displays a prompt icon 10c, indicating that the see-and-speak feature is temporarily unavailable. The mobile phone 100 also displays a prompt message 109, notifying the user that the microphone is occupied by another application and that the see-and-speak feature is temporarily unavailable.

[0203] S1206: The recording management module notifies the monitoring module to monitor the microphone occupancy status, and the monitoring module registers monitoring with the recording module.

[0204] The switch of the see-and-speak function is turned on, the recording management module notifies the monitoring module to monitor the microphone occupancy status, and the monitoring module registers monitoring with the recording module.

[0205] In one implementation, the recording module can monitor the microphone occupancy status based on the activation of AudioRecord. After the monitoring module registers to monitor with the recording module, if the microphone occupancy status changes, the recording module can notify the monitoring module. In one example, if other applications no longer occupy the microphone, the recording module can notify the monitoring module that other applications have stopped occupying the microphone (if no application is occupying the microphone).

[0206] Referring to the example scenarios shown in Figures 9 and 10, in some embodiments, the "See and Speak" feature is turned on, but other applications (such as recording applications, phone applications, audio and video call applications, etc.) are using the microphone to collect user voice, so the "See and Speak" feature is temporarily unavailable. When other applications that are occupying the microphone no longer use it, the "See and Speak" feature is automatically enabled.

[0207] In combination with the software architecture of the terminal device 100 shown in FIG4A , FIG13 shows a flow chart of a method for handling concurrent conflicts in recorded voice provided by an embodiment of the present application in this scenario.

[0208] As shown in FIG13 , the method for handling concurrent conflicts in recorded speech provided in the embodiment of the present application may include:

[0209] S1301: The recording module detects that another application has exited and occupied the microphone.

[0210] After the "See and Speak" function is turned on, the monitoring module registers a monitoring function with the recording module. In this way, when the recording module detects a change in the microphone occupancy status, it can notify the monitoring module.

[0211] In one example, the recording module receives a request from another application to close the recording channel, that is, it monitors that the other application exits and occupies the microphone.

[0212] Optionally, the recording module notifies the ADSP to close the recording channel corresponding to the application.

[0213] S1302: The recording module notifies the monitoring module that other applications have exited and occupied the microphone.

[0214] S1303: The monitoring module notifies the recording management module that other applications have exited and occupied the microphone.

[0215] The recording management module receives a notification that other applications have exited and occupied the microphone, and determines that the switch of the see-and-speak function is turned on (the status management module confirms that the status of the see-and-speak function is temporarily unavailable), and then turns on the see-and-speak function.

[0216] S1304: The recording management module notifies the recording module to establish a recording channel corresponding to the see-and-speak function.

[0217] In one implementation, the recording management module notifies the recording module to create an AudioRecord corresponding to the see-and-speak function.

[0218] S1305. The recording module notifies the ADSP to establish a recording channel corresponding to the see-and-speak function.

[0219] In this way, the audio stream collected by the microphone will be distributed to the See and Talk application.

[0220] S1306. The recording management module notifies the status management module to update the status of the see-and-speak function to enabled, and notifies the service presentation module to issue a prompt on the user interface indicating that the see-and-speak function is enabled.

[0221] S1307. The service presentation module issues a prompt message on the user interface indicating that the "visible and speak" function has been enabled, and displays a prompt icon indicating that the "visible and speak" function has been enabled.

[0222] In one implementation, the service presentation module issues a prompt on the user interface to inform the user that the "See and Speak" feature is enabled. For example, as shown in FIG9(b) and FIG10(b), the status bar of the mobile phone 100 display interface no longer displays prompt icon 10c; instead, the status bar of the mobile phone 100 display interface displays prompt icon 10a, indicating that the "See and Speak" feature is enabled. The mobile phone 100 also displays prompt message 130 to inform the user that the "See and Speak" feature is enabled.

[0223] The method for handling concurrent conflicts in recorded voice provided in an embodiment of the present application, when there is a recording channel of other applications, it means that there are other applications occupying the microphone, and the Visible and Talk function is temporarily unavailable. In some embodiments, when the user's operation to turn on the Visible and Talk function switch is received, it is determined that there are recording channels corresponding to other applications, and the Visible and Talk function is temporarily not turned on. In some embodiments, the Visible and Talk function is turned on, and other applications request to establish corresponding recording channels, then the recording channels corresponding to Visible and Talk are closed. In this way, the audio stream collected by the microphone will be distributed to the application through the recording channels of other applications, and will not be distributed to the Visible and Talk application. This avoids the Visible and Talk being triggered incorrectly.

[0224] When another app exits and occupies the microphone (and the corresponding recording channel for the other app is closed), the See-and-Talk feature automatically starts and establishes a corresponding recording channel for See-and-Talk. This way, the audio stream collected by the microphone is distributed to the See-and-Talk app, allowing users to control the terminal device through voice.

[0225] In the method for handling concurrent voice recording conflicts provided in the embodiments of the present application, after the See-and-Talk feature is turned on, the See-and-Talk application monitors the microphone's occupancy status (when a recording channel is established) and, based on the microphone's occupancy status, controls whether the See-and-Talk feature is enabled or temporarily disabled. This eliminates the need to modify the processing flow of other applications or operating systems, ensuring strong compatibility.

[0226] Below, we'll use a voice assistant application as an example to illustrate the method for handling concurrent voice input conflicts provided by this embodiment. Voice Assistant and See-and-Speak are two voice control functions. In this embodiment, See-and-Speak is referred to as the first voice control function, and Voice Assistant is referred to as the second voice control function.

[0227] In some embodiments, after the "See and Speak" function is turned on, in response to a user operation, the terminal device is awakened and the voice assistant function is turned on. The voice assistant function occupies the microphone, and the "See and Speak" function is temporarily unavailable. After the voice assistant function is turned on, the user can give a command to the terminal device by inputting a voice into the terminal device, and the terminal device executes the command corresponding to the voice. After the terminal device executes a command, or if the terminal device does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after being awakened, the voice assistant function is turned off. The voice assistant function no longer occupies the microphone, and the "See and Speak" function is automatically turned on.

[0228] For example, referring to FIG14(a), the mobile phone 100 has enabled the "See and Speak" feature. For example, the user uses the method shown in FIG2 to turn on the "See and Speak" switch on the mobile phone 100, thereby enabling the "See and Speak" feature. The mobile phone 100 exits the settings application and displays the desktop interface 120. The status bar of the mobile phone 100 display interface displays a prompt icon 10a, indicating that the "See and Speak" feature is enabled. Optionally, the "See and Speak" feature activates the microphone, and the status bar of the mobile phone 100 display interface also displays a prompt icon 10b to remind the user that the microphone is collecting voice.

[0229] The user long-presses the power button of the mobile phone 100. In response to receiving the operation of long-pressing the power button by the user, the mobile phone 100 is awakened. After the mobile phone 100 is awakened, a prompt card is displayed, which indicates that the mobile phone 100 has been awakened and the user can control the mobile phone by voice. The voice assistant function occupies the microphone, and the visible and speaking function is temporarily unavailable. For example, as shown in (b) of Figure 14, the mobile phone 100 displays a voice assistant prompt card, indicating that the mobile phone 100 has been awakened, which is used to prompt the user to input voice into the mobile phone 100 to control the mobile phone 100. The status bar of the display interface of the mobile phone 100 switches from displaying the prompt icon 10a to displaying the prompt icon 10c, indicating that the visible and speaking function is temporarily unavailable.

[0230] For example, as shown in (c) of FIG14 , the user inputs a voice message “open the video” into the mobile phone 100 , and the mobile phone 100 parses and recognizes the user's voice message and executes the instruction corresponding to the voice message “open the video”.

[0231] For example, as shown in FIG14(d), the mobile phone launches the video application in response to receiving the user's voice command "Open video." After executing this command, the mobile phone 100 turns off the voice assistant function. The voice assistant function no longer occupies the microphone, and the see-and-talk function is automatically enabled. The status bar of the mobile phone 100 display interface no longer displays the prompt icon 10c; instead, the status bar displays the prompt icon 10a, indicating that the see-and-talk function is enabled.

[0232] In combination with the software architecture shown in FIG4A , FIG15 shows the interaction process of each module when the voice assistant function and the see-and-speak function are concurrent.

[0233] For example, referring to Figure 15, after the "See and Talk" function is enabled, a recording channel (recording channel 1) corresponding to "See and Talk" is established. The "See and Talk" application also registers a monitor with the recording module.

[0234] S1. The user long-presses the power button of the phone. The voice assistant application receives the user's long-press operation of the power button.

[0235] S2. In response to receiving an operation of the user long pressing the power button, the voice assistant application requests the recording module to establish a recording channel corresponding to the voice assistant function.

[0236] S3. If the recording module notification is visible, it means that the application microphone is occupied by other applications.

[0237] S4. The Visible-Speaking application notifies the recording module to close recording channel 1.

[0238] S5. The recording module notifies ADSP to close recording channel 1 and requests ADSP to re-establish a recording channel.

[0239] ADSP establishes recording channel 2, which is the recording channel corresponding to the voice assistant application.

[0240] Furthermore, after the phone executes a voice command, it turns off the voice assistant function. The voice assistant app requests the recording module to shut down the recording channel corresponding to the voice assistant function. The recording module notifies the ADSP to shut down recording channel 2. The recording module notifies the Visible and Talk app, and other apps to exit the microphone occupation. The Visible and Talk app then notifies the recording module to establish a recording channel corresponding to the Visible and Talk function.

[0241] In this embodiment, after the "See and Speak" feature is activated, a recording channel corresponding to the "See and Speak" feature is established. After the "Voice Assistant" feature is activated, the "Voice Assistant" feature takes over the microphone. The recording channel corresponding to the "See and Speak" feature is closed, and a recording channel corresponding to the "Voice Assistant" feature is established. After the "Voice Assistant" feature is closed, the recording channel corresponding to the "Voice Assistant" feature is closed, and a recording channel corresponding to the "See and Speak" feature is re-established. At any given time, only one recording channel exists, and the audio stream captured by the microphone is distributed to the corresponding application. This prevents system processing errors caused by both the "See and Speak" feature and the "Voice Assistant" feature being enabled.

[0242] In some embodiments, when there is a recording channel, the ADSP in the kernel layer of the terminal device notifies the wake-up word detection module to stop detecting the wake-up word. That is to say, when any application collects voice through the microphone, the terminal device stops detecting the wake-up word. Exemplarily, the mobile phone 100 has turned on the visible and speak function, and the mobile phone 100 displays the desktop interface 120 shown in (a) of Figure 14. The status bar of the display interface of the mobile phone 100 displays a prompt icon 10a, indicating that the visible and speak function is turned on. Optionally, the visible and speak function turns on the microphone, and the status bar of the display interface of the mobile phone 100 also displays a prompt icon 10b to prompt the user that the microphone is collecting voice. The user inputs the voice "Hello YOYO" to the mobile phone 100, and the mobile phone 100 will not be woken up.

[0243] For example, FIG16 is a schematic diagram of the structure of a terminal device 100. The terminal device 100 may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display 294, and a subscriber identification module (SIM) card interface 295. The sensor module 280 may include a temperature sensor, an ambient light sensor, a touch sensor, etc.

[0244] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device 100. In other embodiments, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0245] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0246] The controller can be the nerve center and command center of the terminal device 100. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0247] Processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 210. If processor 210 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 210 latency, and thus improves system efficiency.

[0248] In some embodiments, the processor 210 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0249] It is understood that the interface connection relationship between the modules illustrated in this embodiment is only for illustrative purposes and does not constitute a structural limitation on the terminal device. In other embodiments, the terminal device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0250] The charging management module 240 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 can receive charging input from the wired charger via the USB interface 230. In some wireless charging embodiments, the charging management module 240 can receive wireless charging input via the terminal device's wireless charging coil. While charging the battery 242, the charging management module 240 can also provide power to the terminal device via the power management module 241.

[0251] The power management module 241 is used to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 and provides power to the processor 210, the internal memory 221, the external memory, the display 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 241 can also be set in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

[0252] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0253] The terminal device 100 implements display functions through a GPU, display screen 294, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0254] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0255] The terminal device 100 can realize the shooting function through the ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0256] The ISP processes data fed back by camera 293. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 293.

[0257] The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device may include 1 or N cameras 293, where N is a positive integer greater than 1.

[0258] Digital signal processors are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when a terminal device selects a frequency point, the digital signal processor performs a Fourier transform on the frequency point energy.

[0259] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. This allows the terminal device to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0260] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in terminal devices, such as image recognition, face recognition, speech recognition, and text comprehension.

[0261] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0262] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be set in the processor 210, or some functional modules of the audio module 270 can be set in the processor 210. The speaker 270A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 270B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. The microphone 270C, also known as the "microphone" or "microphone", is used to convert sound signals into electrical signals. The headphone jack 270D is used to connect wired headphones. The headphone jack 270D can be a USB port 230, or it can be a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface. In the embodiment of the present application, after the voice control function is turned on, the terminal device 100 continuously collects surrounding sounds through the microphone 270C.

[0263] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 210 via the external memory interface 220 to implement data storage functions. For example, audio, video, and other files can be stored on the external memory card.

[0264] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the terminal device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 can execute instructions stored in the internal memory 221, and the internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data (such as video files) created during the use of the terminal device, etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0265] The buttons 290 include a power button, a volume button, etc. The button 290 can be a mechanical button. It can also be a touch button. The motor 291 can generate a vibration prompt. The motor 291 can be used for an incoming call vibration prompt, or for touch vibration feedback. The indicator 292 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 295, or pulled out from the SIM card interface 295 to achieve contact and separation with the terminal device. The terminal device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0266] It is understandable that, in order to implement the above functions, the above-mentioned terminal device includes a hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0267] The embodiment of the present application can divide the functional modules of the terminal device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0268] In the case of using integrated units, a possible structural diagram of the terminal device involved in the above embodiment is shown in FIG17 . The terminal device 1600 includes: a processing unit 1601 , a storage unit 1602 , an audio unit 1603 and a display unit 1604 .

[0269] The processing unit 1601 is used to control and manage the actions of the terminal device 1600. For example, it can be used to control the activation, temporary unavailability, and deactivation of the voice control function; it can be used to parse and recognize user voice when it is received, and / or for other processes of the technology described herein.

[0270] The storage unit 1602 is used to store program codes and data of the terminal device 1600. The processing unit 1601 calls the program codes stored in the storage unit 1602 to execute the various steps in the above method embodiment.

[0271] The audio unit 1603 is used to collect the voice of the user and play the voice.

[0272] The display unit 1604 is used to display the interface of the terminal device 1600 .

[0273] Of course, the unit modules in the terminal device 1600 include, but are not limited to, the processing unit 1601, storage unit 1602, audio unit 1603, and display unit 1604. For example, the terminal device 1600 may also include a power supply unit, a communication unit, etc. The power supply unit is used to supply power to the terminal device 1600. The communication unit is used to support communication between the terminal device 1600 and other devices.

[0274] The processing unit 1601 may be a processor or controller, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may include an application processor and a baseband processor. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The storage unit 1602 may be a memory. The audio unit 1603 may include a microphone, a speaker, a receiver, etc. The display unit 1604 may be a display screen, etc. The communication unit may be a transceiver, a transceiver circuit, or a communication interface, etc.

[0275] For example, the processing unit 1601 is a processor (the processor 210 shown in Figure 16), and the storage unit 1602 can be a memory (the internal memory 221 shown in Figure 16). The audio unit 1603 can include a microphone (such as the microphone 270C shown in Figure 16), a speaker (such as the speaker 270A shown in Figure 16), and a receiver (such as the receiver 270B shown in Figure 16). The display unit 1604 can be a display screen (such as the display screen 294 shown in Figure 16). The communication unit includes a mobile communication module (such as the mobile communication module 250 shown in Figure 16) and a wireless communication module (such as the wireless communication module 260 shown in Figure 16). The mobile communication module and the wireless communication module can be collectively referred to as a communication interface. The terminal device 1600 provided in the embodiment of the present application can be the terminal device 100 shown in Figure 16. Among them, the above-mentioned processor, memory, microphone, display screen and communication interface can be coupled together, for example, via a bus connection. The processor calls the program code stored in the memory to execute the various steps in the above method embodiment.

[0276] An embodiment of the present application also provides a chip system (e.g., a system on a chip (SoC)). As shown in FIG18 , the chip system includes at least one processor 1701 and at least one interface circuit 1702. The processor 1701 and the interface circuit 1702 can be interconnected via lines. For example, the interface circuit 1702 can be used to receive signals from other devices (e.g., a memory of a terminal device). For another example, the interface circuit 1702 can be used to send signals to other devices (e.g., the processor 1701 or a touch screen of a terminal device). Exemplarily, the interface circuit 1702 can read instructions stored in the memory and send the instructions to the processor 1701. When the instructions are executed by the processor 1701, the terminal device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete components, which is not specifically limited in the embodiment of the present application.

[0277] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned terminal device, the terminal device executes each function or step in the above-mentioned method embodiment.

[0278] The embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the mobile phone in the above method embodiment.

[0279] Among them, the terminal device 1600, chip system, computer-readable storage medium or computer program product provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0280] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0281] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0282] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0283] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0284] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0285] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for processing concurrent conflicts in recorded voice, applied to a terminal device, wherein the terminal device includes a microphone, and is characterized in that: The method comprises: The terminal device displays a first interface, wherein the first interface includes a first switch; In response to the user turning on the first switch, the terminal device displays a first prompt icon, where the first prompt icon indicates that the first voice control function has been turned on; After receiving the first voice collected by the microphone, the terminal device executes a first instruction corresponding to the first voice; the first voice does not include a wake-up word of the terminal device; In response to the user's operation of turning on the second voice control function, the terminal device stops displaying the first prompt icon, displays the second prompt icon, and displays the first prompt card; the second prompt icon indicates that the first voice control function is temporarily unavailable, and the first prompt card indicates that the second voice control function is turned on; After receiving the second voice collected by the microphone, the terminal device executes the second instruction corresponding to the second voice and stops displaying the first prompt card; After executing the second instruction corresponding to the second voice, the terminal device stops displaying the second prompt icon and displays the first prompt icon.

2. The method according to claim 1, characterized in that: The method further comprises: In response to the user turning on the first switch, the terminal device turns on the recording channel corresponding to the first voice control function, and the recording channel is used to obtain the user voice collected by the microphone.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: In response to the user's operation of turning on the second voice control function, the terminal device turns on the recording channel corresponding to the second voice control function and turns off the recording channel corresponding to the first voice control function.

4. The method according to any one of claims 1 to 3, characterized in that: After executing the second instruction corresponding to the second voice, the method further includes: The terminal device closes the recording channel corresponding to the second voice control function, and opens the recording channel corresponding to the first voice control function.

5. The method according to any one of claims 1 to 4, characterized in that: After the terminal device stops displaying the second prompt icon and displays the first prompt icon, the method further includes: In response to the user's operation of turning on the first function, the terminal device turns on the recording channel corresponding to the first function and turns off the recording channel corresponding to the first voice control function, stops displaying the first prompt icon and displays the second prompt icon; In response to the user's operation of closing the first function, the terminal device closes the recording channel corresponding to the first function and opens the recording channel corresponding to the first voice control function, stops displaying the second prompt icon and displays the first prompt icon.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to the user's operation of turning off the first switch, the terminal device turns off the recording channel corresponding to the first voice control function.

7. The method according to any one of claims 1 to 6, characterized in that: The operation of enabling the second voice control function includes: Long press the power button on the terminal device.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When the terminal device displays the first prompt icon, the terminal device receives a third voice collected by the microphone; the third voice includes a wake-up word of the terminal device; The terminal device continues to display the first prompt icon.

9. The method according to claim 8, characterized in that After the terminal device receives the third voice collected by the microphone, the method further includes: The terminal device obtains the user voice through the recording channel corresponding to the first voice control function.

10. A method for processing concurrent conflicts in recorded voice, applied to a terminal device, the terminal device comprising a microphone, characterized in that: The method comprises: The terminal device displays a first interface, wherein the first interface includes a first switch; In response to the user turning on the first switch, the terminal device turns on the recording channel corresponding to the voice control function and displays a first prompt icon; the recording channel is used to obtain the user voice collected by the microphone, and the first prompt icon indicates that the voice control function is turned on; After receiving the first voice collected by the microphone, the terminal device executes a first instruction corresponding to the first voice; In response to the user's first operation, the terminal device establishes a recording channel corresponding to the first application and closes the recording channel corresponding to the voice control function, stops displaying the first prompt icon and displays a second prompt icon; the second prompt icon indicates that the voice control function is temporarily unavailable.

11. The method according to claim 10, characterized in that The method further comprises: In response to a second operation by the user, the terminal device closes the recording channel corresponding to the first application; In response to the recording channel corresponding to the first application being closed, the terminal device turns on the recording channel corresponding to the voice control function, stops displaying the second prompt icon and displays the first prompt icon.

12. A method for processing concurrent conflicts in recorded speech, applied to a terminal device, the terminal device comprising a microphone, characterized in that: The method comprises: In response to a first operation of the user, the terminal device establishes a recording channel corresponding to the first application; the recording channel is used to obtain the user voice collected by the microphone; The terminal device displays a first interface, wherein the first interface includes a first switch; the first switch is a switch corresponding to a voice control function; receiving an operation of turning on the first switch by a user; In response to the user's operation of turning on the first switch, the terminal device displays a prompt icon, where the prompt icon indicates that the voice control function is temporarily unavailable.

13. The method according to claim 12, characterized in that The method further comprises: In response to a second operation by the user, the terminal device closes the recording channel corresponding to the first application; In response to the recording channel corresponding to the first application being closed, the terminal device opens the recording channel corresponding to the voice control function; After receiving the first voice collected by the microphone, the terminal device executes the first instruction corresponding to the first voice.

14. A terminal device, characterized in that: The terminal device includes: a processor, a memory and a microphone, the microphone is used to collect user voice, and the processor is coupled to the memory; the memory is used to store computer program code; the computer program code includes computer instructions, and when the processor executes the above-mentioned computer instructions, the terminal device executes the method described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 13.

Images