Echo cancellation method, electronic equipment and chip system

Abstract

The invention provides an echo cancellation method, electronic equipment and a chip system, relates to the technical field of signal processing, and is used for improving the echo cancellation capability and the man-machine interaction efficiency. The method comprises the steps that the electronic equipment responds to start a first voice interaction service, and the electronic equipment acquires a first service parameter and acquires a first echo cancellation algorithm corresponding to the first service parameter; and then, the electronic equipment adopts a first echo cancellation algorithm to cancel echoes collected by the electronic equipment through the microphone.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of signal processing technology, and in particular to an echo cancellation method, electronic equipment, and chip system. Background Art

[0002] With the rapid development of voice recognition technology, more and more electronic devices support voice interaction functions. By using the microphone of the electronic device to collect the voice commands issued by the user, human-computer interaction can be carried out with the electronic device, thereby freeing up hands and improving the efficiency of human-computer interaction.

[0003] In some scenarios, the microphone will collect the echo caused by the audio played by the electronic device while collecting the user's voice command, so the collected sound needs to be echo-cancelled. Echo cancellation is the process of eliminating the echo formed in the environment by the audio played by the electronic device itself from the sound collected by the microphone, so that only the user's voice command is retained. Summary of the invention

[0004] The embodiments of the present application provide an echo cancellation method, an electronic device, and a chip system for improving echo cancellation capability and human-computer interaction efficiency.

[0005] The embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an echo cancellation method is provided, which is applied to an electronic device supporting multiple voice interaction services, wherein the multiple voice interaction services include a first voice interaction service; the electronic device includes a microphone and a sound-emitting device; the method includes: in response to the electronic device starting the first voice interaction service, the electronic device obtains parameter information when executing an audio function in a scenario in which the electronic device starts the first voice interaction service, that is, the electronic device obtains the first service parameter.

[0007] Then, the electronic device obtains a first echo cancellation algorithm corresponding to the first service parameter based on the first service parameter, and uses the first echo cancellation algorithm to cancel the echo collected by the electronic device through the microphone.

[0008] It is understandable that the echo is caused by the audio picked up by the microphone and emitted by the sound-generating device.

[0009] In summary, by adopting the solution of the present application, in a scenario where an electronic device starts a first voice interaction service, the electronic device adopts a first echo cancellation algorithm corresponding to the first business parameter based on the first business parameter to eliminate the echo collected by the electronic device through a microphone, that is, the electronic device adapts the first echo cancellation algorithm corresponding to the first business parameter, thereby improving the echo cancellation capability and improving the efficiency of human-computer interaction.

[0010] In a possible implementation of the first aspect, the multiple voice interaction services also include a second voice interaction service; the method also includes: in a scenario where the electronic device switches from the first voice interaction service to the second voice interaction service, in response to the electronic device starting the second voice interaction service, the electronic device obtains parameter information when executing the audio function in the scenario where the electronic device starts the second voice interaction service, that is, the electronic device obtains the second service parameters.

[0011] Then, the electronic device obtains a second echo cancellation algorithm corresponding to the second service parameter based on the second service parameter, and uses the second echo cancellation algorithm to cancel the echo collected by the electronic device through the microphone.

[0012] It should be noted that the first echo cancellation algorithm and the second echo cancellation algorithm have different capabilities.

[0013] In this implementation, in a scenario where the electronic device starts a second voice interaction service, the electronic device uses a second echo cancellation algorithm corresponding to the second service parameters based on the second service parameters to eliminate the echo collected by the electronic device through a microphone; in addition, since the capabilities of the first echo cancellation algorithm and the second echo cancellation algorithm are different, the present application is aimed at obtaining different service parameters for different voice interaction services, and adapting echo cancellation algorithms with different capabilities based on different service parameters, thereby further improving the echo cancellation capability and improving the efficiency of human-computer interaction.

[0014] Optionally, a plurality of echo cancellation algorithms may be stored in the electronic device, and the plurality of echo cancellation algorithms correspond one-to-one to a plurality of voice interaction services; on this basis, the electronic device may select a first echo cancellation algorithm corresponding to the first service parameter from the plurality of echo cancellation algorithms based on the first service parameter; correspondingly, the electronic device may select a second echo cancellation algorithm corresponding to the second service parameter from the plurality of echo cancellation algorithms based on the second service parameter.

[0015] In a possible implementation of the first aspect, the first service parameter includes a service identifier corresponding to the first voice interaction service and a volume type corresponding to the current volume of the electronic device; the electronic device obtains the first service parameter, including: in response to the electronic device starting the first voice interaction service, the electronic device determines the service identifier corresponding to the first voice interaction service; then, the electronic device obtains the volume and distinguishes the volume type corresponding to the volume according to the service identifier.

[0016] In this way, the electronic device can obtain the corresponding first echo cancellation algorithm based on the service identifier, volume and volume type corresponding to the first voice interaction service, and eliminate the echo collected by the microphone, thereby improving the echo cancellation capability and improving the human-computer interaction efficiency.

[0017] In a possible implementation of the first aspect, the first business parameter also includes a current volume of the electronic device; the electronic device uses a first echo cancellation algorithm to eliminate the echo collected by the electronic device through a microphone, including: if the current volume of the electronic device is greater than or equal to a preset volume, the electronic device uses the first echo cancellation algorithm to eliminate the echo collected by the electronic device through the microphone.

[0018] In this way, when the current volume of the electronic device is greater than or equal to the preset volume, the electronic device performs echo cancellation, which can improve the reliability of the echo cancellation.

[0019] Optionally, if the volume of the electronic color code is less than a preset volume, the electronic device does not perform echo cancellation, thereby reducing the power consumption of the electronic device.

[0020] In a possible implementation manner of the first aspect, the method further includes: if the volume of sound changes, the electronic device updates the first service parameter, and determines whether to eliminate the echo collected by the electronic device through the microphone based on the updated first service parameter.

[0021] In this way, when the volume of the electronic device changes, the electronic device can determine whether to eliminate the echo collected by the electronic device through the microphone based on the updated first service parameter, which can further improve the reliability of echo cancellation.

[0022] Optionally, the electronic device may monitor in real time whether the current volume of the electronic device changes, and if the volume changes, update the first service parameter in a timely manner.

[0023] In a possible implementation of the first aspect, an electronic device includes a voice application package APK, and multiple voice interaction services are integrated in the voice APK; wherein the voice APK is a voice interaction service that allows a user to directly speak a voice command without saying a wake-up command, and controls the electronic device to perform an operation corresponding to the voice command; the voice command is a command different from the wake-up command, and the wake-up command is used to wake up an application in the electronic device for voice interaction with the user.

[0024] In a possible implementation manner of the first aspect, the electronic device also includes an audio processor ADSP; and the electronic device obtains the first service parameter, including: the electronic device sends the first service parameter to the ADSP through a voice APK.

[0025] The electronic device acquires the first echo cancellation algorithm corresponding to the first service parameter based on the first service parameter, including: the electronic device matches the first echo cancellation algorithm corresponding to the first service parameter based on the first service parameter through ADSP.

[0026] In a possible implementation manner of the first aspect, the method further includes: the electronic device uses a first echo cancellation algorithm to eliminate environmental noise collected by the electronic device through a microphone.

[0027] In this way, the environmental noise collected by the microphone can be further eliminated by the first echo cancellation algorithm, thereby further improving the echo cancellation capability and improving the human-computer interaction efficiency.

[0028] In a possible implementation of the first aspect, the method further includes: in response to the electronic device initiating any voice interaction service among multiple voice interaction services, the electronic device displays a microphone reception icon; wherein the microphone reception icon is used to represent that the electronic device calls the microphone to collect the user's voice instructions.

[0029] In this way, by displaying the microphone reception icon, the user can be prompted that the microphone of the current electronic device is receiving sound, and the user can directly speak a voice command, thereby improving the user experience.

[0030] In a possible implementation manner of the first aspect, the first voice interaction service is used to implement various operation functions in the third-party application through voice, and the second voice interaction service is used to implement various operation functions in the system application through voice.

[0031] Optionally, the third-party application may be a short video application, a long video application, etc. The system application may be an alarm clock application, a phone application, etc.

[0032] In a possible implementation of the first aspect, in a scenario where the electronic device starts a first voice interaction service, the first echo cancellation algorithm instruction includes: a normalized least mean square adaptive filtering NLMS algorithm and a neural network NN algorithm; in a scenario where the electronic device starts a second voice interaction service, the second echo cancellation algorithm includes at least: an NLMS algorithm and a nonlinear processing NLP algorithm.

[0033] In a possible implementation of the first aspect, the electronic device uses a first echo cancellation algorithm to eliminate the echo collected by the electronic device through a microphone, including: in response to the electronic device starting a first voice interaction service, the electronic device collects first sound information (including a first voice instruction and a first echo) through the microphone; the electronic device uses the first echo cancellation algorithm to eliminate the first echo included in the first sound information, and retains the first voice instruction included in the first sound information; on this basis, the method also includes: the electronic device performs an operation corresponding to the first voice instruction.

[0034] In a second aspect, an electronic device is provided, which has the function of implementing the echo cancellation algorithm described in any one of the first aspects, and the function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0035] In a third aspect, an electronic device is provided, comprising: a microphone, a sound-emitting device, a memory and one or more processors; the memory is used to store computer execution instructions, and when the terminal is running, the processor executes the computer execution instructions stored in the memory to enable the electronic device to perform the echo cancellation method described in any one of the above-mentioned first aspects.

[0036] In a fourth aspect, an electronic device is provided, comprising: a processor; the processor is coupled to a memory, and after reading instructions in the memory, executes the echo cancellation method as described in any one of the above-mentioned first aspects according to the instructions.

[0037] In a fifth aspect, a chip system is provided, which is applied to a terminal. The chip system includes a processor and an interface, wherein the interface is used to receive instructions and transmit them to at least one processor; at least one processor runs the instructions so that the electronic device executes the echo cancellation method as described in any one of the above-mentioned first aspects.

[0038] Optionally, the processor may be an audio processor (ADSP) and / or an application processor (AP).

[0039] In a sixth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer can execute the echo cancellation method described in any one of the above-mentioned first aspects.

[0040] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the echo cancellation method described in any one of the first aspects.

[0041] Among them, the technical effects brought about by any design method in the second to seventh aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1A A schematic diagram of a scenario example of a method for controlling an electronic device by voice provided in an embodiment of the present application;

[0043] Figure 1B A schematic diagram of a scenario example of a method for controlling an electronic device by voice provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of an interface of a voice interaction service scenario provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a voice interaction service scenario provided in an embodiment of the present application Figure 2 ;

[0046] Figure 4 A schematic diagram of a voice interaction service scenario provided in an embodiment of the present application Figure 3 ;

[0047] Figure 5 A schematic diagram of a voice interaction service scenario provided in an embodiment of the present application Figure 4 ;

[0048] Figure 6 A schematic diagram of a voice interaction service scenario provided in an embodiment of the present application Figure 5 ;

[0049] Figure 7 A flowchart of an echo cancellation method provided in an embodiment of the present application is shown in FIG1 ;

[0050] Figure 8 A schematic diagram of a method for echo cancellation provided in an embodiment of the present application Figure 2 ;

[0051] Fig. 9 A schematic diagram of a method for echo cancellation provided in an embodiment of the present application Figure 3 ;

[0052] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0053] Fig.11 A schematic diagram of a software framework of an electronic device provided in an embodiment of the present application;

[0054] Fig.12 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0056] The following terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0057] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0058] Voice control function:

[0059] Many terminal devices support voice control. The terminal device collects user voice through a microphone, analyzes and recognizes the user voice, and executes the instructions corresponding to the user voice, so that the user can control the terminal device through voice.

[0060] Generally speaking, in order to save power consumption of terminal devices and avoid false triggering, the voice control function needs to be turned on before it can be used. For example, the user inputs a preset word (called a wake-up word) to the terminal device through voice to wake up the terminal device. After the terminal device is awakened, it can execute the instructions corresponding to the user's voice, that is, the voice control function is turned on. For example, the user can turn on or off the voice control function by turning on or off the preset switch in the human-computer interaction interface of the terminal device.

[0061] In different terminal devices, the voice control function may have different names, such as "voice control", "intelligent voice", "voice assistant", "see and speak", "voice command", "free command", "intelligent AI", etc. The specific implementation of voice control functions with different names may also be different.

[0062] Several different implementations of the voice control function are described below by way of example.

[0063] Voice Assistant:

[0064] Before using a voice assistant to control an electronic device, the user needs to first input a wake-up word into the electronic device to wake up the electronic device. Generally speaking, before the electronic device is awakened, the microphone of the electronic device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone is only detected for wake-up words at the kernel layer, and the corresponding recording channel of the voice assistant is not started in the system and driver of the electronic device.

[0065] The user inputs a wake-up word to the electronic device through voice. In response to receiving the wake-up word, the electronic device is awakened, and the corresponding recording channel of the voice assistant is started in the system and driver. After the electronic device is awakened, the voice (audio stream) collected by the microphone is sent to the voice assistant application for processing through the recording channel corresponding to the voice assistant. In this way, the electronic device can execute the instructions corresponding to the user's voice, enabling the user to control the electronic device through voice; it can also realize functions such as dialogue with the user.

[0066] For example, Figure 1A The figure shows a schematic diagram of a scenario in which a user uses a voice assistant to control a mobile phone. Figure 1A As shown, the mobile phone displays a desktop interface, and the user inputs a voice "Hello YOYO" into the mobile phone. In response to receiving the wake-up word "Hello YOYO", the mobile phone is awakened. Exemplarily, after the mobile phone is awakened, a voice "I am here" is played to prompt the user that the mobile phone has been awakened.

[0067] After the phone is awakened, the user can control the phone by voice. For example, Figure 1A As shown, the user inputs the voice "open video" to the mobile phone, and the mobile phone analyzes and recognizes the user's voice and executes the command corresponding to the voice "open video". Exemplarily, the mobile phone starts the video application in response to receiving the user's voice "open video".

[0068] In some implementations, after the electronic device is awakened, the user can issue a command to the electronic device by inputting voice into the electronic device, and the electronic device executes the command corresponding to the voice. After the electronic device executes a command, or if the electronic device does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after being awakened, the electronic device no longer responds to the command issued by the user through voice. For example, the electronic device will turn off the recording channel corresponding to the voice assistant. The user needs to input the wake-up word into the electronic device again to wake up the electronic device before he can issue a command to the electronic device by inputting voice into the electronic device again. In other words, after the electronic device is awakened, it enters a "short reception" state and can respond to the user's voice command within a short period of time (for example, within 8 seconds).

[0069] In some implementations, when the electronic device is connected to the network, it supports entering a continuous conversation scenario after being awakened, and the user can have a continuous conversation with the electronic device. After each playback of the electronic device, it will continue to pick up the sound without repeated awakening until the user exits the continuous conversation through instructions such as "exit".

[0070] It can be seen that:

[0071] Visible and Talkable (or Visible and Talkable) is controlled by a preset switch. The user can turn on Visible and Talkable by turning on the preset switch, and can also stop Visible and Talkable by turning off the preset switch. In one implementation, the microphone of the electronic device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. When the preset switch corresponding to Visible and Talkable is turned on, the corresponding recording channel of Visible and Talkable is started in the system and driver of the electronic device. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talkable application for processing through the recording channel corresponding to Visible and Talkable, so that the user can control the electronic device through voice.

[0072] It can be seen that after being turned on, the electronic device enters a "long receiving" state. The user can give instructions to the electronic device through voice at any time without entering a wake-up word to wake up the electronic device.

[0073] Exemplary, reference Figure 1B , the mobile phone displays the desktop interface, the user inputs the voice "open video" into the mobile phone, the mobile phone analyzes and recognizes the user's voice, and executes the command corresponding to the voice "open video". Exemplarily, the mobile phone starts the video application in response to receiving the user's voice "open video".

[0074] It should be noted that it can be said that it is realized locally by the electronic device without the need to connect to the network.

[0075] In the scenario where the user's hands are occupied, voice is the most important and fastest way of human-computer interaction. The electronic device responds to the wake-up command spoken by the user (such as the wake-up word "Hello, YOYO") to achieve interaction between the user and the electronic device. Usually, the electronic device is installed with an application that can interact with the user by voice (such as a voice assistant). The user can interact with the electronic device by voice through the voice assistant to achieve functions that require multiple manual operations to achieve.

[0076] Among them, the scenarios in which the user's hands are occupied mainly include: the user is cooking in the kitchen and using an electronic device to watch videos; or the user is washing and using an electronic device to watch videos or listen to music; or the user is eating on the sofa in the living room and using an electronic device to watch videos; or the user is resting in the bedroom and the electronic device is in the living room and the alarm is ringing.

[0077] In each of the above scenarios, users can use the voice assistant to watch videos (such as pause, play, next / previous, increase / decrease volume, etc.), as well as turn off the alarm / remind later and other operations.

[0078] Currently, when using a voice assistant, the user usually needs to say a wake-up word to wake up the voice assistant. After waking up the voice assistant, the user can continue to say a voice command to control the electronic device to perform an operation corresponding to the voice command. For example, in response to the user saying the wake-up word "Hello, YOYO", the electronic device wakes up the voice assistant. Then, in response to the user saying "Play next", the electronic device can perform the operation of playing the next video.

[0079] In some embodiments, after the electronic device wakes up the voice assistant once and performs an operation corresponding to a voice command, the electronic device automatically turns off the voice assistant. When the user needs to use the voice assistant for a long time for voice interaction, the electronic device needs to frequently respond to the wake-up word spoken by the user to wake up the voice assistant. Frequent wake-up of the voice assistant by the electronic device will inevitably cause higher power consumption, and frequent wake-up of the voice assistant cannot meet the diverse needs of users.

[0080] In some embodiments of the present application, the electronic device includes a voice application package (Android application package, APK), which allows the user to speak a voice command without saying a wake-up command, and directly controls the electronic device to perform a voice interaction service corresponding to the voice command.

[0081] Exemplarily, the voice APK integrates multiple voice interaction services, such as visible and audible services, free command services, artificial intelligence (AI) subtitle services, etc. Among them, the visible and audible services are used to implement various operating functions in third-party applications (such as long video applications, short video applications, etc.) through voice; the free command services are used to implement various operating functions in system applications (such as alarm clock applications, phone applications, etc.) through voice; the AI ​​subtitle service is used to recognize the user's voice and automatically generate synchronized subtitles.

[0082] The following is an exemplary description of the user interface of multiple voice interaction services integrated by the voice APK in the embodiment of the present application in conjunction with the accompanying drawings of the specification. In the following embodiments, multiple voice interaction services including visible and audible services, free command services, and AI subtitle services are used as examples for illustration.

[0083] Among them, multiple voice interaction services have corresponding function entrances in electronic devices, such as a control center, a setting application, etc. Taking the electronic device as a mobile phone and the function entrance as a setting application as an example, in response to the user opening the setting application, the mobile phone displays the following Figure 2 The interface shown in (1) includes a "voice control" setting item. In response to the user's operation on the "voice control" setting item, the mobile phone displays the following Figure 2The interface shown in (2) includes the visible and audible service, the free command service, the AI ​​subtitle service, etc. Of course, the interface can also include other scenario services, such as the free translation service, the on-demand reading service, etc., which are not listed here one by one.

[0084] For example, take the "visible and talkable service" as an example. Figure 2 As shown in (2), in response to the user clicking the “visible and talkable” option, the mobile phone displays the following Figure 2 The prompt page shown in (3) includes prompt information for prompting the user how to use the visible and audible function. Figure 2 The interface shown in (3) also includes a switch for "voice control". When the switch for "voice control" is turned on, the mobile phone starts the visible and talkable function; when the switch for "voice control" is turned off, the mobile phone exits the visible and talkable function. In one example, in response to a user clicking the switch for "voice control", the mobile phone starts the visible and talkable function.

[0085] In some embodiments, after the visible and talkable function is turned on, the mobile phone displays a prompt icon to prompt that the visible and talkable function is turned on. Figure 2 As shown in (4), the status bar of the mobile phone display interface displays a microphone reception icon.

[0086] It should be noted that the activation process of other services (such as free commands, AI subtitles, etc.) can refer to the above-mentioned "Visible and Speakable" activation process, which is similar to the above and will not be listed one by one here.

[0087] In addition, when the mobile phone starts any of the above-mentioned multiple voice interaction services, the status bar of the mobile phone display interface will display the following Figure 2 The microphone receiving icon shown in (4) is used to indicate that the mobile phone is in receiving state, which can prevent the user's privacy from being leaked.

[0088] After the Visible and Talkable function is turned on, the phone enters a continuous receiving state, and the phone's microphone will continue to collect surrounding sounds. Optionally, in one implementation, after any function on the phone turns on the phone's recording channel, the phone will send a prompt message to the user to remind the user that the phone is in the receiving state. In this way, the user's privacy can be prevented from being leaked. For example, after the Visible and Talkable function is turned on, the phone turns on the recording channel and enters a continuous receiving state. For example, Figure 2 As shown in (4), a prompt icon is displayed on the mobile phone display interface, and the prompt icon is used to prompt that the microphone is recording sound.

[0089] Taking the example of starting the visible and talkable service on a mobile phone, for example, after the visible and talkable service is turned on, the mobile phone enters a continuous sound receiving state. The user can input a voice command to the mobile phone, and the mobile phone parses and recognizes the voice command input by the user. If the user's voice command is recognized to match any preset command, the voice command is executed.

[0090] Exemplarily, the preset instructions corresponding to the visible and spoken services include system instructions, short video application instructions, and long video application instructions. The specific instructions may be shown in Table 1 below.

[0091] Table 1

[0092]

[0093] It should be noted that Table 1 is only an example of some preset instructions corresponding to the visible and talkable service in the embodiment of the present application. Of course, the visible and talkable service can also support other preset instructions, which will not be described one by one here.

[0094] For example, when the mobile phone starts the visible and talkable service, in the scenario where the user browses a short video, such as Figure 3 As shown in (1), the mobile phone plays the short video A and displays a prompt message. For example, the mobile phone can display the prompt message in the form of a bubble (such as "You can control the video with language"). Then, in response to the user's voice command (such as "next"), Figure 3 As shown in (2), the mobile phone switches from playing short video A to playing short video B.

[0095] As another example, when the mobile phone starts the visible and talkable service, Figure 4 As shown in (1), the mobile phone displays desktop A and displays prompt information. For example, the mobile phone can display prompt information in the form of bubbles (such as "Please try to speak"). Then, in response to the user's voice command (such as "Swipe left"), Figure 4 As shown in (2), the mobile phone displays desktop B.

[0096] Taking the example of starting the free command service on a mobile phone, the free command service supports system application commands, such as alarm application commands, phone application commands, etc.

[0097] For example, when the mobile phone starts the free command service, when the mobile phone alarm rings, Figure 5 As shown in (1), the mobile phone displays an alarm prompt box in the form of a floating window, and the alarm prompt box includes: the current time (such as 8:00 am), a close control, a later reminder control, etc. In some embodiments, the mobile phone also displays a prompt message in the form of a bubble (such as "Try to say remind me later"). Then, in response to the user's voice command (such as "Remind me later"), Figure 5As shown in (2), the mobile phone exits the alarm prompt box and displays the mobile phone desktop.

[0098] Taking the mobile phone starting the AI ​​subtitle service as an example, the AI ​​subtitle service supports recognizing the user's voice and automatically generating synchronized subtitles. For example, when the mobile phone starts the AI ​​subtitle service, in the scenario where the user is making a video call, Figure 6 As shown, the mobile phone automatically generates synchronized subtitles based on the user's voice during the video call and displays them on the video call interface.

[0099] For example, still Figure 6 As shown, the mobile phone automatically generates synchronized subtitles based on the user's (Bob) voice (such as "Jack, let's go out and play together this weekend") and displays them on the video call interface; correspondingly, the mobile phone automatically generates synchronized subtitles based on the user's (Jack) voice (such as "Sure") and also displays them on the video call interface.

[0100] It should be noted that the visible and speakable service, the free command service and the AI ​​subtitle service include but are not limited to the scenarios listed in the above embodiments, and may also include some other scenarios, which are not limited in the embodiments of the present application.

[0101] In summary, in the embodiments of the present application, by integrating multiple voice interaction services into the same voice APK, the user can be allowed to directly speak a voice command without saying a wake-up word, and the voice APK will control the electronic device to perform the operation corresponding to the voice command. In this way, in scenarios where the user's hands are occupied and voice interaction is required for a long time, the purpose of reducing power consumption can be achieved, and the human-computer interaction experience can be improved.

[0102] It is understandable that in the above scenario, when the electronic device starts any voice interaction service, the microphone of the electronic device is used to collect the voice command issued by the user. While the microphone collects the user's voice command, it also collects the echo caused by the sound device (such as a speaker) of the electronic device playing the audio.

[0103] For example, in the scenario of the visible and speakable service, when a user is browsing a short video, the electronic device collects the user's voice commands through the microphone, and also collects the echo caused by the electronic device's speaker playing the video. In the scenario of the free command service, when the electronic device alarm rings, the electronic device collects the user's voice commands through the microphone, and also collects the echo caused by the electronic device's speaker playing the alarm. In the scenario of the AI ​​subtitle service, when the electronic device is playing audio, the electronic device collects the user's voice through the microphone, and also collects the echo caused by the electronic device's speaker playing audio.

[0104] In order to improve the human-computer interaction experience, it is necessary to perform echo cancellation on the sounds collected by the microphone of the electronic device (including the user's voice and the sounds played by the speakers, such as audio sounds (long video sounds or short video sounds), alarm ringing sounds, incoming call sounds, etc.).

[0105] The embodiment of the present application provides an echo cancellation method, which can use different echo cancellation algorithms for different voice interaction services to perform echo cancellation, thereby improving the echo cancellation capability and the efficiency of human-computer interaction. Specifically, in the scenario where the electronic device starts the first voice interaction service, the first echo cancellation algorithm is used to eliminate the echo caused by the speaker playing audio collected by the microphone; in the scenario where the electronic device starts the second voice interaction service, the second echo cancellation algorithm is used to eliminate the echo caused by the speaker playing audio collected by the microphone.

[0106] Among them, the first voice interaction service and the second voice interaction service can be any one of the visible and speakable service, the free command service, and the AI ​​subtitle service in the above embodiments, and the first voice interaction service and the second voice interaction service are different; the first echo algorithm and the second echo algorithm have different abilities to eliminate echoes.

[0107] The specific process of the echo cancellation method provided in the embodiment of the present application is exemplarily described below with reference to the accompanying drawings of the specification.

[0108] For example, Figure 7 As shown, the electronic device includes a voice APK, an audio digital signal processor (ADSP), and a microphone. The voice APK includes a first voice interaction service and a second voice interaction service; the ADSP includes a first echo cancellation algorithm and a second echo cancellation algorithm; the microphone is used to collect the user's voice command and the echo caused by the speaker playing audio (hereinafter referred to as echo in all embodiments).

[0109] Still Figure 7 As shown, in response to the user's operation on the first voice interaction service, the voice APK starts the first voice interaction service and informs the ADSP that the first voice interaction service is currently started. Correspondingly, after the voice APK starts the first voice interaction service, the microphone collects the first voice command and the first echo, and transmits the collected first voice command and the first echo to the ADSP. Then, the ADSP adopts the first echo elimination algorithm corresponding to the first voice interaction service, eliminates the first echo, retains the first voice command, and reports the first voice command to the voice APK. After receiving the first voice command, the voice APK distributes the first voice command to the corresponding application (application, APPs), so that the application performs the corresponding operation based on the first voice command.

[0110] In some embodiments, after the voice APK starts the first voice interaction service, it sends a first instruction to the ADSP to indicate that the electronic device currently starts the first voice interaction service. In other embodiments, after the voice APK starts the first voice interaction service, it sends a first service parameter corresponding to the first voice interaction service to the ADSP to indicate that the electronic device currently starts the first voice interaction service. Among them, the first service parameter is used to indicate the parameter information when the electronic device executes the audio function in the scenario where the electronic device starts the first voice interaction service.

[0111] Of course, the voice APK can also inform ADSP that the first voice interaction service is currently started in other ways, which are not listed here one by one.

[0112] It should be noted that the voice APK can send the first voice command to the corresponding application through the application framework layer of the electronic device. The above-mentioned application may include: third-party applications, system applications, and desktop applications. The specific implementation process can refer to the description of the following embodiment, which will not be described in detail here.

[0113] Accordingly, in response to the user's operation on the second voice interaction service, the voice APK starts the second voice interaction service and informs the ADSP that the second voice interaction service is currently started. Accordingly, after the voice APK starts the second voice interaction service, the microphone collects the second voice command and the second echo, and transmits the collected second voice command and the second echo to the ADSP. Then, the ADSP adopts the second echo elimination algorithm corresponding to the second voice interaction service, eliminates the second echo, retains the second voice command, and reports the second voice command to the voice APK. After receiving the second voice command, the voice APK distributes the second voice command to the corresponding application, so that the application performs the corresponding operation based on the second voice command.

[0114] Exemplarily, after the voice APK starts the second voice interaction service, it sends the second service parameter corresponding to the second voice interaction service to the ADSP, which is used to indicate that the electronic device currently starts the second voice interaction service. The second service parameter is used to indicate parameter information when the electronic device performs an audio function in the scenario where the electronic device starts the second voice interaction service.

[0115] It should be noted that, for an example of an electronic device initiating a second voice interaction service to perform echo cancellation, reference may be made to the above embodiments, which will not be described in detail here.

[0116] To summarize, in an embodiment of the present application, after the electronic device starts a first voice interaction service, the electronic device may use a first echo cancellation algorithm corresponding to the first voice interaction service to eliminate the echo; after the electronic device starts a second voice interaction service, the second echo cancellation algorithm corresponding to the second voice interaction service may be used to eliminate the echo; since the capabilities of the first echo cancellation algorithm and the second echo cancellation algorithm are different, different echo cancellation algorithms are used for echo cancellation for different voice interaction services, which can improve the echo cancellation capability and human-computer interaction efficiency.

[0117] After the voice interaction service is started with the voice APK, the service parameters corresponding to the voice interaction service (such as the first service parameter or the second service parameter mentioned above) are sent to the ADSP as an example to illustrate the specific process of the echo cancellation method provided in the embodiment of the present application. The service parameters include: one or more of the service identifier, volume, volume type and device type corresponding to the voice interaction service.

[0118] Exemplary, combined Figure 7 ,like Figure 8 As shown, the voice APK includes a service processing module, a service presentation module, a data acquisition module, and an echo cancellation management module. The ADSP includes an audio conversion module and an echo cancellation algorithm module. The electronic device also includes a system APK, which includes a volume module.

[0119] like Figure 8 As shown, the business processing module is connected to the business presentation module and the data acquisition module respectively, the data acquisition module is connected to the echo cancellation management module and the volume module respectively; the echo cancellation algorithm module is connected to the echo cancellation management module and the audio conversion module respectively; the audio conversion module is connected to the microphone.

[0120] Among them, the business processing module is used to obtain the business identifier corresponding to the voice interaction business. Exemplarily, in response to the user's operation of starting the voice interaction business, the business processing module distinguishes the business identifier corresponding to the voice interaction business through the acoustic echo cancellation (AEC) mode. For example, when the electronic device starts the visible and speakable service, the business processing module distinguishes the business identifier of the visible and speakable service as 1 through AEC_mode; when the electronic device starts the free command service, the business processing module distinguishes the business identifier of the free command service as 2 through AEC_mode; when the electronic device starts the AI ​​subtitle service, the business processing module distinguishes the business identifier of the AI ​​subtitle service as 3 through AEC_mode.

[0121] It should be noted that the service identifiers shown in the above embodiments are merely examples of the embodiments of the present application and do not constitute a limitation on the present application. Of course, service identifiers may also be represented in other ways (such as A, B, C), which are not listed here one by one.

[0122] The service presentation module is used to display a microphone reception icon. Exemplarily, in response to a user's operation of starting a voice interaction service, the service processing module notifies the service presentation module to display a microphone reception icon in the status bar of the electronic device.

[0123] The data acquisition module is used to obtain the service identification from the service processing module and obtain the current volume of the electronic device from the volume module. The data acquisition module distinguishes the volume type corresponding to the volume according to the service identification. For example, when the service identification is 1, the data acquisition module distinguishes the volume type corresponding to the volume as the media volume (i.e., media); when the service identification is 2, the data acquisition module distinguishes the volume type corresponding to the volume as the alarm volume (i.e., alarm); when the service identification is 3, the data acquisition module distinguishes the volume type corresponding to the volume as the incoming call volume (i.e., ring). The data acquisition module is also used to obtain the device type of the electronic device (such as iPad, phone, etc.).

[0124] Then, the data acquisition module sends the service parameters (such as service identification, volume, volume type and device type) to the echo cancellation management module. The echo cancellation management module is used to manage the service parameters corresponding to the voice interaction service and send the service parameters to the echo cancellation algorithm module.

[0125] The audio conversion module is used to obtain the user's voice command and echo from the microphone, and convert the obtained voice command and echo into electrical signals (such as voice signals and echo signals). Then, the audio conversion module sends the voice signal and echo signal to the echo cancellation algorithm module.

[0126] The echo cancellation algorithm module matches the echo cancellation algorithm corresponding to the business parameters based on the business parameters, eliminates the echo signal, and retains the voice signal.

[0127] In summary, by adopting the solution of the embodiment of the present application, after starting the voice interaction service, the electronic device can obtain the service parameters corresponding to the voice interaction service, and then match the echo cancellation algorithm corresponding to the service parameters based on the service parameters to eliminate the echo signal and retain the voice signal, thereby improving the echo cancellation capability and human-computer interaction efficiency.

[0128] In some embodiments, after the echo cancellation algorithm module obtains the service parameters, the echo cancellation algorithm module determines whether echo cancellation is required based on the volume included in the service parameters. Exemplarily, if the volume is greater than or equal to a preset volume threshold, the echo cancellation algorithm module determines that echo cancellation is required; if the volume is less than the preset volume threshold, the echo cancellation algorithm module determines that echo cancellation is not required.

[0129] In this way, when the volume is greater than or equal to the preset volume, the echo cancellation algorithm module performs echo cancellation, and when the volume is less than the preset volume, it means that the audio played by the speaker of the electronic device is low, so the echo cancellation algorithm module may not perform echo cancellation, thereby achieving the purpose of reducing power consumption.

[0130] In some embodiments, the data acquisition module registers a callback with the volume module for calling back the volume status of the electronic device. Exemplarily, after the volume module detects that the volume of the electronic device has changed, it calls back the changed volume to the data acquisition module, and the data acquisition module determines whether echo cancellation is required based on the changed volume. If necessary, the data acquisition module re-acquires the service parameters and sends the re-acquired service parameters to the echo cancellation management module.

[0131] In this way, the echo cancellation management module can match the service parameters re-issued by the data acquisition module with the echo cancellation algorithm corresponding to the service parameters to eliminate the echo, which is beneficial to improving the reliability of echo cancellation.

[0132] Exemplary, combined Figure 8 ,like Fig. 9 As shown, the voice APK also includes a data processing module; the ADSP also includes a playback module and a shared memory. The electronic device also includes a speaker. Please refer to Fig. 9 , the data processing module is connected to the shared memory; the playing module is connected to the audio conversion module and the speaker respectively.

[0133] The playback module is used to output an audio signal to the speaker, which converts the audio signal into audio and plays it. It is understandable that the audio played by the speaker will cause an echo after being received by the microphone. Therefore, when the microphone collects the user's voice command, it will also collect the echo caused by the audio played by the speaker.

[0134] The shared memory is used to store the voice signal after the echo cancellation algorithm module eliminates the echo, and sends the voice signal to the data processing module. The data processing module is used to convert the voice signal into a corresponding voice command and send the voice command to the corresponding application.

[0135] In some embodiments, Fig. 9As shown, the echo cancellation algorithm module includes multiple echo cancellation algorithms, and the multiple echo cancellation algorithms correspond to multiple voice interaction services one by one. For example, echo cancellation algorithm 1 corresponds to the visible and speakable service, and echo cancellation algorithm 2 corresponds to the free command service.

[0136] For example, the echo cancellation algorithm module is preset with a correspondence between the echo cancellation algorithm and the service parameters, wherein the correspondence can be stored in the echo cancellation algorithm module in the form of a table or an array, and the correspondence can be shown in Table 2, for example. Table 2 illustrates the service parameters including service identifier, volume type, and device type as an example.

[0137] Table 2

[0138] Service parameters (service identifier, volume type, device type) Echo Cancellation Algorithm (1, media, phone) Echo Cancellation Algorithm 1 (2. alarm, phone) Echo Cancellation Algorithm 2 …… ……

[0139] It should be noted that Table 2 is only an example of the correspondence between the echo cancellation algorithm and the service parameters in the embodiment of the present application. Of course, the echo cancellation algorithm module may also include correspondences between other service parameters and echo cancellation algorithms, which are not listed one by one here.

[0140] For example, Fig. 9 As shown, in response to the user's operation of starting the voice interaction service, the electronic device calls the microphone to collect the user's voice command. In the scenario where the electronic device starts the voice interaction service, when the electronic device needs to play audio, the playback module outputs an audio electrical signal to the speaker, and the speaker converts the audio electrical signal into audio and plays it. In this way, while collecting user instructions, the microphone also collects the echo caused by the audio played by the speaker, that is, the microphone can collect voice instructions and echoes.

[0141] Then, the microphone transmits the voice command and the echo to the audio conversion module, which converts the voice command into a voice signal and the echo into an echo signal. Correspondingly, the playback module also transmits the audio electrical signal as a reference signal to the audio conversion module. Furthermore, the audio conversion module sends the voice signal, the echo signal, and the reference signal to the echo cancellation algorithm module.

[0142] Accordingly, in response to the user's operation of starting the voice interaction service, the service processing module obtains the service identifier corresponding to the voice interaction service, and sends the service identifier to the data acquisition module. Then, the data acquisition module obtains the current volume of the electronic device, and distinguishes the volume type corresponding to the volume according to the service identifier; the data acquisition module is also used to obtain the device type of the electronic device. Further, the data acquisition module sends the service identifier, volume, volume type and device type as service parameters to the echo cancellation management module. The echo cancellation management module sends the service parameters to the echo cancellation algorithm module.

[0143] In some embodiments, in combination with the above Table 2, if the service parameters include 1, media, and phone, the echo cancellation algorithm module selects echo cancellation algorithm 1 to eliminate the echo signal.

[0144] Exemplarily, the echo cancellation algorithm 1 includes: front-end gain control, normalized least mean square adaptive filtering (NLMS), neural network (NN), dereverberation, and automatic gain control (AGC).

[0145] Among them, the front-end gain control is used to increase the gain of the signal in order to reduce the difficulty of subsequent signal processing.

[0146] NLMS is used for echo cancellation. Exemplarily, NLMS uses an adaptive filtering algorithm to adjust the weight vector of the filter, estimates an approximate echo path to approximate the real echo path, thereby obtaining an estimated echo signal, and uses a microphone to collect a mixed signal (speech signal + echo signal) minus the estimated echo signal to achieve echo cancellation. For example, with 10ms as a block, the data of the reference signal is divided into multiple blocks, and multiple NLMS filters are used to learn the echo of each block respectively and superimpose them to obtain the weight vector of the filter.

[0147] After the NLMS is used to perform echo cancellation, there will still be a residual echo signal (i.e., a residual signal). Exemplarily, the residual signal satisfies the following expression: e(n) = d(n) - w(n) T*x(n) ; e(n) represents the residual signal, d(n) represents the speech signal and echo signal output by the audio conversion module, w(n) represents the reference signal, and T represents the weight vector.

[0148] NN is used to suppress the residual signal output by NLMS to enhance the effect of echo cancellation. Exemplarily, the residual signal output by NLMS, and the speech signal and echo signal output by the audio conversion module are used as inputs of NN, and the nonlinear characteristics of the echo signal are learned by neural network model training, thereby suppressing the nonlinear echo signal.

[0149] For example, the neural network model can be a convolutional neural network (CNN), a deep neural network (DNN), a gated recurrent unit (GRU), or a combination of CNN+GRU. Of course, the neural network model can also be other models, which are not described here. As long as the effect of suppressing the echo signal can be achieved, it should belong to the protection scope of the embodiments of the present application.

[0150] De-reverberation is used to suppress reverberation in directions other than the incident direction of the voice command to the microphone. Since reverberation is caused by multiple reflections of the sound source, it can be considered that the reverberation of the current frame is generated by the convolution of the previous frames of voice through the room impulse response, so an adaptive filtering algorithm can be used for single-channel dereverberation.

[0151] Automatic gain control (AGC) is used to control the volume gain of a speech signal. Exemplarily, an AGC algorithm is used to adjust the level of the speech signal to a preset target level.

[0152] In some other embodiments, as shown in Table 2 above, if the service parameters include 2, alarm, and phone, the echo cancellation algorithm module selects echo cancellation algorithm 2 to eliminate the echo signal. Exemplarily, echo cancellation algorithm 2 includes: front-end gain control, NLMS, non-linear processing (NLP), dereverberation, and AGC.

[0153] It can be understood that, different from the above-mentioned echo cancellation algorithm 1, the echo cancellation algorithm 2 uses NLP to suppress the residual signal output by NLMS after using NLMS to perform echo cancellation, so as to enhance the effect of echo cancellation.

[0154] Exemplarily, NLP is used to construct a gain expression to estimate the residual signal based on the similarity (cohxd) between the speech signal and the echo signal output by the audio conversion module, and the similarity (cohxe) between the residual signal and the reference signal.

[0155] For example, the residual signal satisfies the following expression: hnl=alpha*cohxd+(1-alpha)*cohxe; hnl represents the residual signal, alpha represents the smoothing weighting coefficient, cohxd represents the similarity between the speech signal and the echo signal, and cohxe represents the similarity between the residual signal and the reference signal.

[0156] The gain satisfies the following expression: gain=max(min_gain, 1.0-hnl); gain represents the gain, and hnl represents the residual signal.

[0157] It should be noted that, for the examples of front-end gain control, NLMS, dereverberation and AGC included in the echo cancellation algorithm 2, reference may be made to the above embodiments, which will not be described one by one here.

[0158] Still Fig. 9 As shown, after the echo cancellation algorithm module eliminates the echo signal, the voice signal is retained and stored in the shared memory. Then, the data processing module reads the voice signal from the shared memory, converts it into a voice command, and sends it to the corresponding application so that the application performs the corresponding operation according to the voice command.

[0159] Exemplarily, the data processing module includes a process of converting a voice signal into a voice command, including: voice recognition, intent understanding, and dialogue management. Among them, voice recognition is used to convert a voice signal into text information; intent understanding is used to convert text information into semantics that can be understood by the electronic device, and understand the intent of the text information based on the semantics; dialogue management is used to generate a voice command that the electronic device finally executes based on the understood intent of the text information.

[0160] For example, speech recognition can be achieved through the automatic speech recognition (ASR) function in electronic devices; intent understanding can be achieved through the natural language understanding (NLU) function in electronic devices; and dialogue management can be achieved through the dialogue management (DM) function in electronic devices.

[0161] In summary, when the echo cancellation module adopts different echo cancellation algorithms for echo cancellation, since different echo cancellation algorithms use different algorithms for suppressing echoes, the final echo cancellation capabilities are different. Therefore, adopting different echo cancellation algorithms for different voice interaction services can improve the echo cancellation capability and human-computer interaction efficiency.

[0162] In some embodiments of the present application, the echo cancellation algorithm can also eliminate the environmental noise collected by the microphone. The specific implementation process of eliminating the environmental noise can refer to the relevant technology, which will not be described here one by one.

[0163] It should be noted that the contents recorded in each embodiment of the present application can explain and illustrate the technical solutions in other embodiments of the present application, and the technical features recorded in each embodiment can also be applied in other embodiments and combined with the technical features in other embodiments to form a new solution. The present application only exemplarily lists several embodiments for illustration, and does not mean that the present application is limited to this.

[0164] The embodiment of the present application provides an electronic device, which may include: a memory and one or more processors; the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device performs various functions or steps performed in the above electronic device. The structure of the electronic device can refer to the structure of the following electronic device 100.

[0165] like Fig.10 As shown, it is a structural schematic diagram of an electronic device 100 provided in an embodiment of the present application. Among them, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a positioning module 181, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0166] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0167] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor (Modem), a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), an ADSP, a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0168] In the embodiment of the present application, the AP is used to determine the service parameters corresponding to the voice interaction service and send the corresponding service parameters to the ADSP. After receiving the service parameters sent by the AP, the ADSP matches the corresponding echo cancellation algorithm based on the service parameters to perform echo cancellation.

[0169] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0170] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0171] In some embodiments, the processor 110 may include one or more interfaces. It is understood that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0172] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 charges the battery 142, it can also power the electronic device through the power management module 141.

[0173] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0174] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0175] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0176] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-OLED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc.

[0177] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170 (such as the above-mentioned audio conversion module), the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.

[0178] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. The speaker 170A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. The microphone 170C, also known as the "microphone" or "microphone", is used to convert sound signals into electrical signals. The headphone jack 170D is used to connect wired headphones.

[0179] In the embodiment of the present application, when the electronic device starts the voice interaction service, the microphone 170C is used to collect the user's voice commands; while the microphone 170C collects the user's voice commands, it also collects the echo caused by the speaker 170A playing the audio.

[0180] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, audio, video and other files are stored in the external memory card.

[0181] The internal memory 121 may be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 121. For example, in an embodiment of the present application, the processor 110 may execute the instructions stored in the internal memory 121, and the internal memory 121 may include a program storage area and a data storage area.

[0182] The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0183] Exemplarily, the processor 110 can execute computer executable program codes stored in the internal memory 121, and the executable program codes include instructions. The processor 110 executes the instructions to perform various functional applications and data processing of the electronic device. For example, in an embodiment of the present application, the AP in the processor 110 executes the instructions stored in the internal memory, executes the corresponding steps, and sends a message to the network side through the modem.

[0184] The buttons 190 include a power button, a volume button, etc. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0185] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM card, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0186] In addition, on top of the above components, the electronic device 100 also runs an operating system, such as an iOS operating system, an Android operating system, etc. Application programs can be installed and run on the operating system.

[0187] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplarily illustrate the software structure of the electronic device 100.

[0188] Fig.11 It is a software structure block diagram of the electronic device according to an embodiment of the present application.

[0189] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer (framework), the Android runtime (Android runtime) and system library, the hardware abstraction layer (HAL), and the kernel layer (kernel).

[0190] The application layer may include a series of application packages APK. For example, the application layer may include voice APK, etc. The voice APK includes a UI layer and a logic layer; the UI layer includes a service presentation module. The service presentation module is used to display a microphone icon when the voice APK of the electronic device starts the voice interaction service;

[0191] The logic layer includes a service processing module, a data acquisition module, an echo cancellation management module and a data processing module. The service processing module is used to determine the service identifier corresponding to the voice interaction service and send the service identifier to the data acquisition module in the logic layer. The data acquisition module is used to obtain the current volume of the electronic device and distinguish the volume type corresponding to the volume according to the service identifier. The data acquisition module is also used to obtain the device type of the electronic device.

[0192] The echo cancellation management module is used to determine the service parameters according to the service identifier, volume, volume type and device type, and call the capability connection interface to send the service parameters to the ADSP. Among them, the capability connection interface is the connection interface between the software layer and the hardware of the electronic device.

[0193] The data processing module is used to convert the voice signal after echo cancellation into voice commands, and call the software development kit (SDK) to send the voice commands to the application framework layer (such as the multi-modal interaction framework). Fig.11 As shown, the process of converting voice signals into voice commands by the data processing module includes: voice recognition, intent understanding, and dialogue management.

[0194] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0195] like Fig.11 As shown, the application framework layer may include a multi-modal interaction framework and a page perception framework. Among them, the page perception framework is used to determine the application currently running on the electronic device based on the page currently displayed by the electronic device. The page perception framework sends the application identifier (such as application package name, application name, etc.) of the currently running application to the multi-modal interaction framework, and the multi-modal interaction framework calls the SDK to send the voice command to the corresponding application.

[0196] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0197] The core library consists of two parts, one is the function that Java voice needs to call, and the other is the Android core library.

[0198] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0199] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0200] HAL can include multiple library modules, each of which implements a set of interfaces for a specific type of hardware component. For example, HAL includes display HAL, camera HAL, audio HAL, and sensor HAL. The kernel layer is the layer between hardware and software. The kernel layer includes display drivers, camera drivers, audio drivers, sensor drivers, etc.

[0201] In some embodiments of the present application, Fig.11 As shown, the electronic device also includes an ADSP, which includes an audio conversion module, an echo cancellation algorithm module, and a shared memory. The audio conversion module is used to obtain voice commands and echoes collected by a microphone, convert the voice commands into voice signals, and convert the echoes into echo signals. The echo cancellation algorithm module matches the corresponding echo cancellation algorithm based on business parameters, eliminates the echo signal, and retains the voice signal. The shared memory is used to store the voice signal.

[0202] Exemplarily, the ADSP saves the voice signal after echo elimination in a shared memory, and the data processing module in the voice APK reads the voice signal from the shared memory and converts the voice signal into a voice command.

[0203] Then, the data processing module calls the SDK to send the voice command to the multimodal interaction framework. The multimodal interaction framework obtains the application name (such as package name) of the application currently running on the electronic device and sends the voice command to the corresponding application so that the application performs the corresponding operation based on the voice command.

[0204] The present application also provides a chip system, such as Fig.12 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 may be Fig.10The processor 110 is shown. The interface circuit 1102 may be, for example, an interface circuit between the processor 110 and an external memory; or an interface circuit between the processor and the internal memory 121 .

[0205] The processor 1101 and the interface circuit 1102 can be interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (such as the memory of the electronic device 100). For another example, the interface circuit 1102 can be used to send signals to other devices (such as the processor 1101). Exemplarily, the interface circuit 1102 can read the instructions stored in the memory and send the instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can execute the various steps performed by the electronic device in the above embodiment. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiments of the present application.

[0206] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes each function or step executed by the electronic device in the above method embodiment.

[0207] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the electronic device in the above method embodiment.

[0208] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0211] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0213] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An echo cancellation method, characterized in that: Applicable to an electronic device supporting multiple voice interaction services, wherein the multiple voice interaction services include a first voice interaction service; the electronic device includes a microphone and a sound generating device; and the method includes: In response to the electronic device starting the first voice interaction service, the electronic device acquires a first service parameter; wherein the first service parameter is used to indicate parameter information when the electronic device performs an audio function in a scenario where the electronic device starts the first voice interaction service; The electronic device acquires, based on the first service parameter, a first echo cancellation algorithm corresponding to the first service parameter; The electronic device uses the first echo cancellation algorithm to cancel the echo collected by the electronic device through the microphone; the echo is caused by the audio collected by the microphone and emitted by the sound-generating device.

2. The method according to claim 1, characterized in that The multiple voice interaction services also include a second voice interaction service; and the method further includes: In a scenario where the electronic device switches from the first voice interaction service to the second voice interaction service, in response to the electronic device starting the second voice interaction service, the electronic device acquires a second service parameter; the second service parameter is used to indicate parameter information when the electronic device performs an audio function in the scenario where the electronic device starts the second voice interaction service; The electronic device acquires, based on the second service parameter, a second echo cancellation algorithm corresponding to the second service parameter; The electronic device uses the second echo cancellation algorithm to cancel the echo collected by the electronic device through the microphone; The first echo cancellation algorithm and the second echo cancellation algorithm have different capabilities in canceling the echo.

3. The method according to claim 1 or 2, characterized in that: The first service parameter includes a service identifier corresponding to the first voice interaction service and a volume type corresponding to the current volume of the electronic device; The electronic device acquires a first service parameter, including: In response to the electronic device starting the first voice interaction service, the electronic device determines a service identifier corresponding to the first voice interaction service; The electronic device obtains the volume, and distinguishes a volume type corresponding to the volume according to the service identifier.

4. The method according to claim 3, characterized in that The first service parameter also includes a current volume of the electronic device; and the electronic device uses a first echo cancellation algorithm to cancel the echo collected by the electronic device through the microphone, including: If the current volume of the electronic device is greater than or equal to the preset volume, the electronic device uses a first echo cancellation algorithm to cancel the echo collected by the electronic device through the microphone.

5. The method according to claim 4, characterized in that The method further comprises: If the volume changes, the electronic device updates the first service parameter, and determines whether to eliminate the echo collected by the electronic device through the microphone based on the updated first service parameter.

6. The method according to any one of claims 1 to 5, characterized in that The electronic device includes a voice application package APK, and the multiple voice interaction services are integrated in the voice APK; Among them, the voice APK allows the user to directly speak a voice command without saying a wake-up command, and controls the electronic device to perform an operation corresponding to the voice command. The voice command is a command different from the wake-up command, and the wake-up command is used to wake up the application in the electronic device for voice interaction with the user.

7. The method according to claim 6, characterized in that The electronic device further includes an audio processor ADSP; and the electronic device acquires the first service parameter, including: The electronic device sends the first service parameter to the ADSP through the voice APK; The electronic device acquiring, based on the first service parameter, a first echo cancellation algorithm corresponding to the first service parameter, includes: The electronic device matches the first echo cancellation algorithm corresponding to the first service parameter based on the first service parameter through the ADSP.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: The electronic device uses the first echo cancellation algorithm to eliminate the environmental noise collected by the electronic device through the microphone.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: In response to the electronic device starting any one of the multiple voice interaction services, the electronic device displays a microphone receiving icon; The microphone sound collection icon is used to indicate that the electronic device calls the microphone to collect the user's voice command.

10. The method according to any one of claims 2 to 9, characterized in that: The first voice interaction service is used to implement various operation functions in third-party applications through voice, and the second voice interaction service is used to implement various operation functions in system applications through voice.

11. The method according to claim 10, characterized in that In the scenario where the electronic device starts the first voice interaction service, the first echo cancellation algorithm at least includes: a normalized least mean square adaptive filtering NLMS algorithm and a neural network NN algorithm; In a scenario where the electronic device starts the second voice interaction service, the second echo cancellation algorithm includes at least: the NLMS algorithm and a non-linear processing NLP algorithm.

12. The method according to any one of claims 1 to 11, characterized in that The electronic device uses a first echo cancellation algorithm to cancel the echo collected by the electronic device through the microphone, including: In response to the electronic device starting the first voice interaction service, the electronic device collects first sound information through the microphone; wherein the first sound information includes a first voice command and a first echo; The electronic device uses the first echo cancellation algorithm to eliminate the first echo included in the first sound information, and retains the first voice instruction included in the first sound information; Wherein, the method further comprises: The electronic device performs an operation corresponding to the first voice instruction.

13. An electronic device, characterized in that: The electronic device comprises: a microphone, a sound generating device, a memory and one or more processors; The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 12.

14. A chip system, characterized in that: Applied to electronic devices, the chip system includes: at least one processor and an interface, the interface is used to receive instructions and transmit them to the at least one processor; the at least one processor executes the instructions so that the electronic device executes the method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 12.

Images