Voice interaction method and electronic equipment

Abstract

The invention provides a voice interaction method and electronic equipment, and the method comprises the steps: enabling the power consumption of the electronic equipment to be increased when the electronic equipment is lifted in a screen-off state, and enabling the power consumption of the electronic equipment to be reduced to a value before the power consumption is increased when the electronic equipment is still in the screen-off state and the lifting state is changed to be not lifted. According to the scheme, in the voice interaction process under the breath wake-up scene, when the electronic equipment is lifted and then the lifting is ended, the power consumption is reduced in time, so that unnecessary power consumption caused by hand lifting without breath wake-up is eliminated, and the overall power consumption is reduced.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a voice interaction method and electronic device. Background Art

[0002] When users perform voice interaction on electronic devices such as mobile phones, tablets, smart wearable devices, car computers, smart speakers, etc., the traditional method is to say a specific wake-up word recorded in advance to wake them up. Users need to remember the different wake-up words for different electronic devices, and they cannot wake them up if they say the wrong ones. In order to solve this problem, the breath wake-up method has emerged. In the breath wake-up method, users only need to get close to the microphone of the electronic device with the screen off or on and speak the interactive voice. They no longer need to wake up with the wake-up word and then speak the interactive voice. However, since breath wake-up requires the electronic device to continue working even when the screen is off and the application processor (AP) is in sleep mode, this leads to higher power consumption requirements.

[0003] In order to reduce power consumption, a solution has emerged to deploy the breath wake-up module in the low-power space of the audio digital signal processor (ADSP), so that the ADSP can continuously detect breath wake-up in low-power mode and wake up the AP for voice interaction when the breath wake-up function is triggered. However, since only the low-power space of high-end (high-configuration) ADSPs has enough low-power space to achieve the above deployment, and the low-power memory space of more ADSPs is not enough to deploy the complete breath wake-up module, the above solution has poor universality. Under the premise of giving priority to universality, a solution has emerged to divide the breath wake-up module into functional parts and deploy them in the low-power space and non-low-power space of ADSP respectively, so that the ADSP can work in low-power mode most of the time when the screen is turned off, and work in normal working mode under certain trigger conditions to complete breath wake-up detection and wake up the AP. However, compared with the solution that is fully deployed in the low-power space of ADSP, this increases power consumption.

[0004] Therefore, how to reduce the power consumption of breath awakening as much as possible while ensuring the universality of the breath awakening solution is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a voice interaction method and electronic device, which can effectively reduce the power consumption of breath awakening while ensuring the universality of the breath awakening solution.

[0006] In a first aspect, a method for voice interaction is provided, which is applied to an electronic device, and the method includes: at a first moment, the electronic device is lifted up and the screen of the electronic device is in an off state, and the power consumption of the electronic device changes from a first value to a second value, before the first moment, the screen of the electronic device is in an off state, the electronic device is not lifted up, and the second value is greater than the first value; at a second moment after the first moment, the electronic device is not lifted up and the screen of the electronic device is in an off state, and the power consumption of the electronic device changes from the second value to the first value.

[0007] In the technical solution of the present application, during the voice interaction in the breath awakening scenario, when the electronic device is lifted and then ends being lifted, the power consumption is reduced back to low power in time, thereby eliminating unnecessary power consumption caused by raising the hand for non-breath awakening, so as to reduce the overall power consumption.

[0008] In combination with the first aspect, in certain implementations of the first aspect, at a second moment after the first moment, the electronic device is not lifted, and the screen of the electronic device is in an off-screen state, and when the power consumption of the electronic device changes from a second value to a first value, it may include: at the second moment, based on the electronic device changing from a lifted state to a not lifted state, the power consumption of the electronic device changes from the second value to the first value. In this implementation, it is explained why the power consumption value changes at the second moment, because the electronic device changes based on the change in the lifted state in the off-screen state, and it can also be understood that the power consumption change at the second moment occurs under the triggering of the change in the lifted state from lifted to not lifted, and combined with the above first moment, which is when the lifted state of the electronic device changes from not lifted to lifted, it constitutes a complete solution for changing the power consumption based on the change in the lifted state of the electronic device. In the traditional scheme, there will not be a second moment like the present application (or it can be understood that the second moment will not be like the present application where the electronic device will end the process based on the change in the lifting state). In the traditional scheme, once the electronic device is lifted and the power consumption changes from the first value to the second value, it is necessary to wait for at least the third moment below (no suitable voice data is collected within the preset time period) or the sixth moment (the entire voice interaction process is completed) before returning to the first value. Therefore, compared with this, the present application has the effect of "stopping losses in time", and can end the process in time after the lifting state changes, without wasting power consumption by executing subsequent processes in vain.

[0009] In combination with the first aspect, in some implementations of the first aspect, the above method also includes: before the first moment, the first processor of the electronic device continuously detects hand-raising events in a low power consumption mode, and the hand-raising events are used to characterize whether the electronic device is lifted up, wherein when the hand-raising event is detected, it indicates that the electronic device is lifted up, and when the non-hand-raising event is detected, it indicates that the electronic device is not lifted up; at the first moment, in response to detecting the hand-raising event, the first processor switches from the low power consumption mode to the normal working mode, and begins to determine whether a breath wake-up event is triggered in the normal working mode, and continues to obtain the detection results of the hand-raising event in the normal working mode; at the second moment, in response to detecting the non-hand-raising event, the first processor switches from the normal working mode to the low power consumption mode, and continuously detects hand-raising events in the low power consumption mode. In this implementation, the first processor continuously detects hand-raising events in low power consumption mode. Once a hand-raising event is detected, it indicates that the electronic device has been lifted up, that is, the first moment arrives. The electronic device switches from low power consumption to normal working mode in response to detecting the hand-raising event, so that the power consumption increases from the previous first value to the subsequent second value starting from the first moment. From the first moment to the second moment, the first processor is in normal working mode until the second moment when the electronic device detects a non-hand-raising event again, and thus knows that the lifting state has ended. Based on this, the first processor is directly returned to the low power consumption mode, instead of waiting for a sufficient period of time and executing subsequent processes, thereby effectively reducing power consumption.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: between the first moment and the third moment, the electronic device remains lifted up, and the screen of the electronic device is in an off-screen state, the time interval between the first moment and the second moment is less than the time interval between the first moment and the third moment, and the time interval between the first moment and the third moment is a preset time interval; at the third moment, the power consumption of the electronic device changes from the second value to the first value. In this implementation, the step of maintaining a fixed waiting time of the traditional solution is continued to be retained, so that the timely termination of the process can be ensured on the basis of superimposing the solution of the present application, that is, if the second moment does not occur, that is, the lifting state of the electronic device does not change, then the process is terminated after the preset time interval is counted from the first moment, and the power consumption is reduced back to the first value.

[0011] In one example, at the third moment, the power consumption of the electronic device changes from the second value to the first value, including: at the third moment, based on the electronic device not collecting voice data that meets the first preset condition, the power consumption of the electronic device changes from the second value to the first value, and the first preset condition includes: the signal strength of the voice data collected by the first microphone of the electronic device is greater than or equal to the first preset signal strength threshold, the signal strength of the voice data collected by the second microphone of the electronic device is greater than or equal to the second preset signal strength threshold, and the difference between the signal strength of the voice data collected by the first microphone and the signal strength of the voice data collected by the second microphone is greater than or equal to the preset signal strength difference threshold. In this example, the triggering factor for adjusting the power consumption at the third moment is given, that is, based on the fact that the electronic device does not collect voice data that meets the first preset condition. In combination with the above, there are two ways in which the present application can trigger "returning to low power consumption", the lifting state is changed (or understood as the electronic device ending the lifting state) and the voice data that meets the first preset condition is not collected within the preset time interval. The two triggering methods can be executed separately or superimposed. Only executing the first method is the present application solution, and only executing the second method is the traditional solution. The best effect of reducing power consumption is to execute both methods.

[0012] In one example, at the third moment, based on the fact that the electronic device does not collect voice data that meets the first preset condition, the power consumption of the electronic device changes from the second value to the first value, including: before the first moment, the first processor of the electronic device operates in a low power consumption mode; between the first moment and the third moment, the first processor operates in a normal working mode; the first processor continuously reads the voice data collected by the first microphone and the second microphone from the first moment; at the third moment, the first processor determines that the breath awakening event is not triggered based on the fact that the electronic device is lifted and the electronic device does not collect voice data that meets the first preset condition, controls the first processor to switch from the normal working mode to the low power consumption mode, and continuously detects the hand raising event in the low power consumption mode. In this example, it is mainly explained that the numerical change of power consumption is caused by the change of the working mode of the first processor. Before the first moment, the first processor is in a low power consumption mode, so the power consumption is the lowest first value, between the first moment and the third moment is a normal working mode, so the power consumption is a higher second value, at the third moment, because no voice data that meets the first preset condition is collected, the breath awakening event is not triggered, so that the first processor is put into the low power consumption mode again, so the whole process shows that the power consumption changes from the first value to the second value and then back to the third value.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: at a fifth moment after the first moment, the electronic device turns on the screen, and the power consumption of the electronic device is a third value, and the third value is greater than the second value. In this implementation, the power consumption change caused by the successful breath awakening is mainly given.

[0014] In one example, at a fifth moment after the first moment, the screen of the electronic device is turned on, and the power consumption of the electronic device is a third value, which is greater than the second value, including: at the first moment, the first processor of the electronic device is switched from a low power consumption mode to a normal working mode, thereby increasing the power consumption of the electronic device from the first value to the second value; at a fourth moment after the first moment, based on the electronic device starting to collect voice data that meets the first preset condition, the first processor starts to continuously read the collected voice data; the first preset condition includes: the signal strength of the voice data collected by the first microphone of the electronic device is greater than or equal to the first preset signal strength threshold, and the signal strength of the voice data collected by the second microphone of the electronic device is greater than or equal to the first preset signal strength threshold. The degree is greater than or equal to the second preset signal strength threshold, and the difference between the signal strength of the voice data collected by the first microphone and the signal strength of the voice data collected by the second microphone is greater than or equal to the preset signal strength difference threshold; at the fifth moment after the fourth moment, the first processor has finished reading the first voice data, and the first processor wakes up the second processor based on the first voice data satisfying the second preset condition, so that the second processor switches from the low power consumption mode to the normal working mode and performs voice recognition and response to the first voice data in the normal working mode, thereby increasing the power consumption of the electronic device from the second value to the third value, and satisfying the second preset condition is used to indicate that the first voice data can be used to trigger the breath awakening event. In this example, it is given how the power consumption changes from the first moment to the fifth moment change with the working mode of the processor. Before the first moment, the first processor and the second processor are both in low power consumption mode, between the first moment and the fourth moment, the first processor is in normal working mode, between the fourth moment and the fifth moment, the first processor is in normal working mode, and after the fifth moment, the second processor and the first processor are both in normal working mode, so that when the awakening is successful, the power consumption continues to increase.

[0015] In another example, the method further includes: at a fifth moment, based on the first voice data not satisfying the second preset condition, the first processor switches from the normal working mode to the low power consumption mode, so that the power consumption of the electronic device changes from the second value to the first value. In this example, when the first processor determines that the breath wake-up event is not triggered, it switches back to the low power consumption mode.

[0016] In another example, the method further includes: at a sixth moment after the fifth moment, the screen of the electronic device changes from the on state to the off state, and the power consumption of the electronic device changes from the third value to the first value. In this example, when the screen of the electronic device turns off again, the power consumption returns to the first value.

[0017] In another example, at a sixth moment after the fifth moment, the screen of the electronic device changes from the on state to the off state, and the power consumption of the electronic device changes from the third value to the first value, including: at the sixth moment, based on the second processor completing the response to the first voice data, the first processor and the second processor both switch to a low power consumption mode, so that the power consumption of the electronic device changes from the third value to the first value. In this example, after the second processor completes the response, both processors are switched to a low power consumption mode, thereby reducing the power consumption to the first value.

[0018] In another example, the method further includes: at the fourth moment, based on the electronic device not collecting voice data that meets the first preset condition, the first processor does not read the voice data that meets the first preset condition, and the electronic device continues to collect voice data until the third moment, and the time interval between the third moment and the first moment is the preset time interval. In this example, if the voice data has not been collected at the fourth moment, the voice data will continue to be collected until the third moment arrives.

[0019] In a second aspect, a voice interaction device is provided, which includes a unit composed of software and / or hardware for executing any one of the methods in the first aspect.

[0020] In a third aspect, an electronic device is provided, which includes a memory, multiple processors, a sensor, an audio acquisition device and a display screen, wherein the sensor is used to collect motion data of the electronic device, the audio acquisition device is used to collect voice of the electronic device, the display screen is used to display an interface, the memory is used to store instructions, and the multiple processors are used to run the instructions stored in the memory, so that the electronic device executes any one of the methods in the first aspect.

[0021] In a fourth aspect, a system is provided, which is applied to an electronic device, wherein the chip system includes a first processor and a second processor, the first processor being used to perform breath wake-up detection, and the second processor being used to respond to voice interaction to voice data collected by the second processor when awakened by the first processor; at a first moment, the electronic device is lifted up, and the screen of the electronic device is in an off state, and the first processor triggers the power consumption of the electronic device to change from a first value to a second value, and before the first moment, the screen of the electronic device is in an off state, the electronic device is not lifted up, and the second value is greater than the first value; at a second moment after the first moment, the electronic device is not lifted up, and the screen of the electronic device is in an off state, and the power consumption of the electronic device changes from the second value to the first value.

[0022] Optionally, the chip also includes a memory, and the memory is electrically connected to the processor.

[0023] Optionally, the chip may also include a communication interface.

[0024] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by an electronic device, any one of the methods in the first aspect can be implemented.

[0025] In a sixth aspect, a computer program product is provided, which includes a computer program, and when the computer program is executed by an electronic device, any one of the methods in the first aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1-Figure 3 It is a schematic interaction diagram of a voice interaction method in an embodiment of the present application.

[0027] Figure 4 It is a schematic diagram of the software architecture of an electronic device according to an embodiment of the present application.

[0028] Figure 5 It is a schematic diagram of the execution process of the breath awakening module during a voice interaction process in an embodiment of the present application.

[0029] Figure 6 It is a schematic flowchart of a voice interaction method of an embodiment of the present application.

[0030] Figure 7 It is a schematic diagram of the execution process of breath awakening detection in an embodiment of the present application.

[0031] Figure 8 It is a schematic flow chart of a speech recognition stage of an embodiment of the present application.

[0032] Fig. 9 It is a schematic flowchart of a dialogue stage of an embodiment of the present application.

[0033] Fig.10 This is a comparison chart of power consumption of the embodiment of the present application and the traditional solution in the same interaction scenario.

[0034] Fig.11 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] The scheme of the embodiment of the present application is introduced below with reference to the accompanying drawings.

[0036] Voice interaction is a human-computer interaction method. Electronic devices can perform corresponding actions based on the voice commands input by the user, such as performing information query, music playback, starting a certain function, etc. If the user wants to interact with an electronic device by voice, he needs to wake up the voice interaction module of the electronic device first. Ways to wake up the voice interaction module may include: shortcut key wake-up, wake-up word wake-up method and breath wake-up method. The wake-up word wake-up method requires the user to enter a specific wake-up word. The breath wake-up method allows the user to wake up the device or application by making certain voices (such as voice commands, breathing sounds, coughing sounds or other non-specific vocabulary sounds) without entering a specific wake-up word. Among them, the breath wake-up method can also be called wake-up word-free wake-up, and this application does not specifically limit its name.

[0037] The breath awakening method requires continuous detection or periodic checking of signals (such as voice signals) to determine whether to wake up the voice interaction module. When the breath awakening method requires the participation of the main processor such as the application processor (AP), even low-frequency periodic checks may frequently wake up the main processor, which conflicts with the sleep state of the main processor, resulting in increased power consumption of the electronic device. In order to achieve the situation where the screen of the electronic device is turned off (the screen of the electronic device is off), that is, the main processor (such as AP) enters the sleep state, the voice interaction module is awakened by the breath awakening method, and the power consumption of the electronic device is low, the embodiment of the present application deploys the breath awakening module used to achieve breath awakening in the low-power space of other processors (non-main processors), such as low-power memory space. The design of the low-power space allows specific tasks to continue to be processed when other parts of the system (such as AP) are in sleep mode, such as processing audio data or sensor data, thereby reducing the power consumption of the electronic device.

[0038] For example, the other processor may be a coprocessor, or an auxiliary processor, which can continue to work regardless of whether the main processor (AP) is in sleep mode. For example, the coprocessor may be an audio digital signal processor (ADSP) or a system control processor (SCP). The coprocessor may also be a sensor hub chip. When the user performs voice wake-up, the other processor may wake up the AP to perform subsequent operations.

[0039] When the low-power memory space of the ADSP of the chip platform is large enough, the breath wake-up module can be fully deployed in the low-power memory space of the ADSP. However, if the low-power memory space of the ADSP of the chip platform is insufficient, the breath wake-up module cannot be fully deployed in the low-power memory space of the ADSP. In other words, due to the limited low-power memory space of the ADSP, the breath wake-up method is difficult to universalize, that is, it is difficult to extend the application to electronic devices with too small low-power memory space. Taking mobile phones as an example, mobile phones with high chip configurations generally use relatively high ADSP configurations and sufficient low-power space, and can deploy complete breath wake-up modules without pressure, but other mobile phones with ordinary chip configurations cannot be deployed. This undoubtedly leads to the breath wake-up solution can only be applied to high-configuration mobile phones, and cannot be extended to all mobile phones. In order to solve this problem, a solution has emerged in which the breath wake-up module is divided into functions, and part of it is deployed in the low-power memory space, while the rest is deployed in the non-low-power space of the ADSP. In this solution, the gesture detection module in the low-power memory space of ADSP is mainly used to continuously perform gesture detection in the low-power working mode when the screen is off, and only after detecting the hand-raising gesture will the relevant modules in the non-low-power memory space of ADSP be awakened to perform the relevant steps of breath awakening detection, including collecting voice signals and judging whether to wake up the main processor AP, otherwise it will return to the low-power working mode and continue to perform gesture detection in the low-power mode. The above-mentioned step-by-step awakening of ADSP and AP allows the breath awakening solution to be deployed on electronic devices with mid- and low-end chips (small low-power memory space), improving the universality of breath awakening.

[0040] However, the breath wake-up solution developed to improve universality still increases the extra power consumption to a certain extent. That is, ADSP needs to perform breath wake-up detection and identification in normal working mode (not low-power working mode). In particular, in order to ensure the accuracy of detection, it needs to maintain a sufficiently long time after waking up ADSP to ensure accurate collection of audio signals, identification and awakening of AP, which makes the power consumption of the waiting time indispensable.

[0041] In order to further reduce power consumption while ensuring the universality of the breath wake-up solution, the present application solution was born. The present application solution mainly found through analysis that some false wake-ups caused unnecessary power consumption of ADSP. For example, the user may flip the electronic device over, put down the electronic device immediately after picking it up, or pick up the electronic device but not bring it close to the mouth, etc., and the lifting gesture will be detected, thereby waking up the ADSP, and it is necessary to wait for a sufficient period of time to collect audio signals and make subsequent judgments. In fact, in the above scenarios, users do not want to be awakened by breath. By analyzing the commonalities of these erroneous wake-up situations, a solution was found, that is, to "correct" the detection results of the hand-raising gesture, thereby eliminating and promptly terminating the subsequent steps of the above-mentioned erroneous wake-up, so as to effectively reduce power consumption, thus giving rise to the present application solution.

[0042] To facilitate understanding of the present application scheme, the above-mentioned low-power mode, sleep state, etc. are illustrated below. Taking the main processor as AP as an example, the AP entering the sleep state can be understood as the AP entering the low-power mode, which can reduce energy consumption and extend the use time of the electronic device. When the AP enters the sleep state, it includes but is not limited to the following situations: 1. The clock frequency of the AP is reduced or stopped. The clock is the basis for the processor to execute instructions. Reducing or stopping the clock frequency can reduce the power consumption of the processor. 2. In order to match the reduced clock frequency, the power supply voltage of the AP is reduced accordingly. 3. Turn off or suspend some circuits, such as cache, graphics processor, etc.

[0043] For electronic devices with indicator lights, when the AP enters the sleep state, the indicator lights of the electronic devices may flash in a specific manner or remain on / off. For electronic devices with display screens, when the AP enters the sleep state, the screen of the electronic devices may turn off or display a specific sleep screen saver.

[0044] Taking the main processor as AP as an example, all operations performed by the AP can be implemented through the main processor. The situation where the main processor enters a sleep state or enters a low-power mode may include the situation where the above-mentioned AP enters a sleep state or enters a low-power mode. The embodiment of the present application is only illustrated by taking AP as an example, and does not mean that the present application is limited to this.

[0045] The electronic device provided in the embodiment of the present application has a voice interaction module. For example, the electronic device is installed with a voice assistant (such as YOYO, etc.) application that can provide a voice interaction function. The user wakes up the voice assistant to achieve voice interaction between the user and the electronic device. Waking up the voice assistant installed in the electronic device is to wake up the voice interaction module of the electronic device.

[0046] The "voice assistant" involved in the embodiments of the present application may also be referred to as a "smart assistant", "digital assistant", "virtual assistant", "intelligent automated assistant" or "automatic digital assistant", etc. "Voice assistant" can be understood as an information processing system that can recognize natural language input in voice form and / or text form to infer user intent and perform corresponding actions based on the inferred user intent. The system can output a response to the user's input in an audible (e.g., voice) and / or visual form.

[0047] The electronic device provided in the embodiment of the present application has a breath wake-up module for waking up the voice interaction module. For example, the voice assistant includes a voice wake-up module and a breath wake-up module. When the voice wake-up module is turned on, the user can wake up the smart assistant by using a wake-up word. The electronic device wakes up the smart assistant in response to receiving a specific wake-up word. When the breath wake-up module is turned on, the user can wake up the smart assistant without using a wake-up word, and the user can wake up the smart assistant by using a breath wake-up method. In response to the electronic device being in a raised state or in response to a hand-raising event, the electronic device receives audio data sent by the user, and wakes up the smart assistant based on the audio data.

[0048] Among them, the breath awakening module can also be called a "wake-up word-free awakening module" or a "direct dialogue awakening module", etc., and this application does not make any specific limitations on this.

[0049] Figure 1-Figure 3 is a schematic interaction diagram of a voice interaction method in an embodiment of the present application. Figure 1 As shown, a desktop interface 101 is displayed on the electronic device, and the desktop interface includes icons of multiple application programs (applications, apps), such as icons of a phone app, a contact book app, a settings app, and a message app. However, it should be understood that the present application does not limit the number of icons and specific icons included in the interface 101, as well as the positions of the icons in the interface.

[0050] The electronic device can be a mobile phone, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), home appliance, smart wearable device, etc. This application does not impose any restrictions on the specific type of electronic device, but requires that the user can perform a lift gesture operation on the electronic device, and the electronic device can detect the lift gesture, and the electronic device also needs to support intelligent voice interaction. The electronic device can be a device running an Android system, an IOS system, a Windows system or other operating systems. This application does not limit the type of electronic device that executes the voice interaction method provided in this application and the operating system run by the electronic device, as long as it can be used to execute the voice interaction method of this application.

[0051] Assume that a user clicks on the icon of the setting app in interface 101, and the electronic device responds to the click operation by displaying interface 102, which is the main interface of the setting app. Interface 102 includes option bars for multiple services, such as WLAN, Bluetooth, mobile network, notification and status bar, desktop and wallpaper, and smart assistant services.

[0052] Assume that the user clicks on the smart assistant option bar (option B) in interface 102, and the electronic device responds to the click operation and displays interface 103, which is the main interface of the smart assistant. Interface 103 includes the wake-up word "Hello, YOYO", as well as controls (option bars) for various smart services and various smart interactions. Smart services may include YOYO suggestions, negative one screen, smart vision, etc., which are mainly some services that can use various models to provide users with smart services. Smart interaction may include smart voice, which can mainly give smart feedback during human-computer interaction. The smart voice control here is the management option for voice interaction corresponding to this application.

[0053] Assume that the user clicks on the smart voice option bar (control C) in interface 103, and the electronic device responds to the click operation and displays interface 104. Interface 104 is a management interface for smart voice. As shown in interface 104, the interface includes a human-computer interaction diagram of smart voice interaction, as well as a functional explanation of smart voice "through natural voice dialogue, help you use the device efficiently and answer your questions", and also includes various controls, each of which corresponds to a specific type of smart voice. Here, interface 104 includes voice wake-up controls, breath wake-up controls, button wake-up controls, desktop shortcut controls, and broadcast tone controls as an example. In practice, other controls may also be included, or some of the above controls may be reduced, without limitation.

[0054] Assume that the user clicks the breath awakening control (control D) in the interface 104, and the electronic device responds to the click operation by displaying Figure 2 Interface 105 is shown.

[0055] Interface 105 is a management interface for breath wake-up, which includes instructions, operation prompts, switch controls (i.e., control E), etc. Among them, the operation prompts can show the user how to wake up the smart assistant through the breath wake-up function through animation. The operation prompts can also include text prompts, such as "natural conversation, instant questions and answers, no need to say the "Hello YOYO" wake-up word". The instructions for breath wake-up are used to introduce the operation method of the breath wake-up method. The instructions may include, for example, text prompts: lift the phone, bring the bottom of the phone close to your mouth (within 7 cm), and speak voice commands directly to the bottom microphone, such as "What's the weather like today". As a result, when the breath wake-up module is running, the user can wake up the smart assistant by directly outputting voice commands without having to output the wake-up word first. It should be understood that the above is only an example of interface 105, and there is no specific limitation. For example, the distance close to the mouth can be other distance thresholds such as 5 centimeters (cm) or 6 centimeters. As long as the difference in the strength of the audio signals received by the microphones at the top and bottom of the electronic device is large enough when the user speaks into the electronic device, and the signal strength of the audio signal received by the microphone close to the mouth is large enough, other situations will not be listed one by one.

[0056] As shown in interface 105, the initial state of the switch control (control E) of breath awakening is off. Assuming that the user clicks control E in interface 105, the electronic device responds to the click operation and displays interface 106. It can be seen that the other contents in interface 106 are the same as those in interface 105, except that the state of control E is on, thereby turning on the breath awakening function.

[0057] It should be understood that Figure 1-Figure 2 This is mainly an example of how to enable the breath wake-up function in the Settings app, but in practice the specific interaction process can also be similar to Figure 1-Figure 2 Not exactly the same. For example, when the smart assistant service is created as a desktop shortcut icon or desktop card, the user can directly enter the smart assistant interface 103 by clicking the above desktop shortcut icon or desktop card. Other situations are not listed one by one.

[0058] After the breath wake-up function is turned on in an electronic device, the user can directly wake up the smart voice assistant by breath wake-up when the electronic device is off, so as to interact with the voice. Figure 3 For example, it should be understood that Figure 1-Figure 2The steps of setting the on and off status of the breath awakening function do not have to be performed before each breath awakening, but each breath awakening can be performed under the premise that the breath awakening function is turned on.

[0059] Figure 3 It is a schematic interaction diagram of a voice interaction method according to an embodiment of the present application. Figure 3 The process shown needs to be carried out under the premise that the breath awakening function of the electronic device is turned on.

[0060] like Figure 3 As shown in (a), the electronic device is in a screen-off state, the AP processor is in a sleep state, and a microphone is provided on the top of the electronic device ( Figure 3 The bottom is provided with another microphone ( Figure 3 However, it should be understood that the bottom microphone may be represented by microphone M and the top microphone may be represented by microphone N, and there is no limitation thereto.

[0061] Assume that the user lifts the electronic device so that the microphone N at the bottom is close to the user's mouth. In response to the lifting gesture, the electronic device starts to collect the voice signal input by the user. The lifting gesture can be collected by an accelerometer, a gyroscope and / or an inertial measurement unit (IMU) in the electronic device.

[0062] The distance between the electronic device and the user's mouth can be maintained at 0-7 cm, which makes it easy for the microphone M and microphone N of the mobile phone to accurately collect the user's audio data.

[0063] from Figure 3 The gesture state shown in (a) is Figure 3 The gesture state shown in (b) is an example of a hand-raising operation, that is, the user can raise the electronic device by raising the wrist gesture, so that the electronic device is closer to the user's mouth. The motion data of the electronic device that can be collected by the sensor in the electronic device may include acceleration, tilt, impact, vibration, rotation and other data of the electronic device. Figure 3 As shown in (a) and (b), the electronic device is lifted up, the electronic device is currently in a hand-held lifted state (i.e., lifted state), or a hand-lifting event occurs on the electronic device, wherein the hand-held lifted state of the electronic device can be understood as the user holding the electronic device in his hand. It should be understood that Figure 3(a) and (b) are just examples of the lift gesture. There is no limitation on the actual lift gesture scenario, as long as the lift gesture brings the electronic device close to the user's mouth. Since breath wake-up requires close proximity to the microphone to speak, no matter what the initial state of the electronic device is, there must be a lift movement process, which brings the microphone of the electronic device close to the user's mouth. Therefore, the electronic device only needs to detect the posture change to determine whether the lift gesture has occurred.

[0064] like Figure 3 As shown in (b), it is assumed that the user brings the microphone N close to his mouth through a lifting gesture and then says the voice "How long have I exercised today", and the electronic device responds to the voice input, lights up the screen and displays interface 107. It should be understood that the breath wake-up is performed in the screen-off state, so after detecting the lifting gesture, the electronic device needs to first collect voice through two microphones, and then determine whether breath wake-up is required based on the voice, and when it is determined that breath wake-up is required, wake up the AP processor, turn on the screen and make subsequent responses.

[0065] Since the microphone M and the microphone N are located at different positions on the electronic device, and the distance between the microphone M and the sound source (the user's mouth) is different from the distance between the microphone M and the sound source (the user's mouth), the signal strength of the voice received by the microphone M and the microphone N may be different, and there may be an energy difference between the voice signals received by the microphone M and the microphone N. When the user speaks to the top of the electronic device, the voice energy of the voice collected by the top microphone M will be higher than the voice energy of the voice collected by the bottom microphone N. Similarly, when the user speaks to the bottom of the electronic device, the voice energy of the voice collected by the bottom microphone N will be higher than the voice energy of the voice collected by the top microphone M.

[0066] It should also be understood that it is possible to speak close to a microphone while making a phone call, but this situation occurs during the call, during which the AP is in a working state after being awakened (not in a sleep state), and the electronic device will not detect and respond to breath awakening, so this situation does not fall into the scenario involved in this application.

[0067] ADSP determines whether the breath wake-up event is triggered. When it is detected that the breath wake-up event is triggered, ADSP wakes up the AP, and the AP starts or resumes the smart assistant. When the breath wake-up event is not triggered, ADSP does not wake up the AP, nor does it start or resume the smart assistant.

[0068] For example, when the electronic device is lifted, the electronic device collects the corresponding voice of the user when speaking through microphones M and N, as well as the motion data collected by the sensor, and sends the collected voice and motion data to ADSP. ADSP detects the input voice data and motion data through the breath wake-up module to determine whether the breath wake-up event is triggered.

[0069] like Figure 3 As shown in the middle interface 107, when a breath wake-up event is detected based on the input voice data and / or motion data, the ADSP wakes up the AP, and the electronic device lights up and displays the smart assistant wake-up interface 107. The wake-up interface 107 includes text content converted from the input voice data and a control F that prompts the user that the smart assistant has been started. Figure 3 As shown in (b), the audio data input by the user is “How long did I exercise today?”, and the corresponding text content in the interface 107 is “How long did I exercise today?” The user can click control F to trigger the smart assistant to end the voice recognition.

[0070] In some embodiments, the wake-up interface 107 may also include the interface content before the smart assistant is awakened, for example, the interface 107 also includes part of the content of the interface 106.

[0071] After identifying the text content of the voice data input by the user, the electronic device can respond. Here, since "How long have I exercised today" is an intelligent question and answer for the sports health app, the electronic device can respond to the text content, schedule the intelligent body of the sports health app and its corresponding question and answer large model to infer the corresponding answer and display it, as shown in interface 108. The user's question "How long have I exercised today" and the corresponding answer "You have walked for 15 minutes and run for 30 minutes today, a total of 45 minutes of exercise. From the sports health app". It should be understood that interface 108 is only an example of how to respond after voice recognition. In practice, it can also be presented in other forms or other corresponding responses, and there is no limitation. For example, interface 108 may not include the character image of the breath awakening demonstration, and for example, it may not be in the form of a dialogue bubble, but in the form of a list. For another example, assuming that the question is not for the sports health app, such as a question corresponding to other apps such as weather, other content can be displayed in other forms. For another example, assuming that the user inputs "Help me open the gallery", the interface 108 may be displayed as "Help me open the gallery", and the feedback "OK, I will help you check in the gallery app", and automatically jump to the display interface of the gallery app after the display is completed, or, under this premise, the interface 108 is not displayed, but the running interface of the gallery app is directly displayed, and the voice broadcast or light prompt "OK, I have helped you check in the gallery". For another example, assuming that the user inputs "Call Zhang San", the interface 108 may be displayed as "Call Zhang San", and the feedback "OK, I have helped you find Zhang San's phone number 123456 from the contacts, and I will open the phone app", and automatically jump to the display interface of the phone app and display Zhang San's phone number, or, under this premise, the interface 108 is not displayed, but the running interface of the phone app is directly displayed and Zhang San's phone number is displayed, and the voice broadcast "OK, I have helped you enter Zhang San's phone number, please confirm whether to call the number, thank you". Other situations are not listed one by one.

[0072] When the ADSP confirms that the smart assistant needs to be awakened, the ADSP generates a wake-up signal and transmits the wake-up signal to the AP. For example, the ADSP detects that a breath awakening event is triggered, and the ADSP generates a wake-up signal. For another example, the ADSP detects a wake-up word, and the ADSP generates a wake-up signal. The ADSP can send a wake-up signal to the AP through the communication interface within the system, such as I2C, SPI, or a dedicated wake-up line.

[0073] If the AP was previously in a low-power mode such as a sleep state, it will return to a normal working mode during the wake-up process. Exemplarily, after the AP receives the wake-up signal, the AP will wake up or configure necessary system resources, including but not limited to: adjusting the power management policy, increasing the clock frequency, enabling necessary hardware interfaces, etc. For example, after the AP's power management unit (power management unit, PMU) recognizes the wake-up signal, the PMU changes the AP's power state and provides the AP with the required voltage and current, such as activating a power converter, adjusting the voltage level, or increasing the supply current. For another example, the AP's clock management unit (clock management unit, CMU) adjusts the clock frequency.

[0074] After the AP receives the wake-up signal, the AP checks the status of the smart assistant and performs corresponding processing according to the status of the smart assistant.

[0075] If the smart assistant has not been started, the AP starts the smart assistant service. The AP starting the smart assistant service may include but is not limited to: initializing resources, loading services, starting services, user interface preparation, service and component initialization. Exemplarily, initializing resources may include but is not limited to: the AP allocates necessary resources to the smart assistant service, including central processing unit (CPU) time slices, memory, and possible input and output (I / O) resources. Exemplarily, loading services may include but is not limited to: the AP will load the code and resources of the service from storage into memory. Exemplarily, starting services may include but is not limited to: the AP starts the entry point of the smart assistant service, such as the main Activity or a specific startup Service. Exemplarily, user interface preparation may include but is not limited to: the AP will prepare and display the user interface so that the user can see the status and feedback of the service.

[0076] If the smart assistant is already running in the background, but the smart assistant is in a dormant or low-power state, the AP wakes up the smart assistant and restores the smart assistant to a normal working state. The AP waking up the smart assistant and restoring the smart assistant to a normal working state may include but is not limited to: waking up the smart assistant, allocating computing resources, resuming services, and updating the user interface. Exemplarily, waking up the smart assistant may include but is not limited to: the operating system converts the state of the smart assistant from a dormant or low-power state to a normal working state. Exemplarily, allocating computing resources may include but is not limited to: the AP reallocates necessary computing resources, such as CPU and memory, to ensure that the smart assistant can operate normally. Exemplarily, restoring services may include but is not limited to: the AP restarts services that have been stopped due to low power consumption. Exemplarily, updating the user interface may include but is not limited to: if the smart assistant has a user interface, the AP will ensure that the interface can be displayed normally after waking up, and update any pending notifications or information. The AP restores the interactive functions between the user and the smart assistant, such as clicking, sliding, voice input, etc., so that the user can continue to use the smart assistant for various operations.

[0077] If some functions of the smart assistant are running in the background, the AP restores the complete functions of the smart assistant, including but not limited to: user interface display, background services. Exemplarily, restoring the main interface display may include but not limited to: the main interface of the smart assistant (such as the negative one screen, sidebar, etc.) is displayed normally on the screen, restoring the interactive functions between the user and the smart assistant, such as clicking, sliding, voice input, etc., and restoring data synchronization, such as user preferences, settings, history records, etc. Exemplarily, restoring various services of the smart assistant running in the background may include but not limited to: notification reminders, health monitoring, security protection, etc. It can be understood that the above description takes the voice interaction module as an example of the smart assistant. After the ADSP wakes up the AP, the AP can refer to the above wake-up voice interaction module. If the voice interaction module has not been started, the AP starts the voice interaction module. The AP starts the voice interaction module and may include but is not limited to: initializing resources, loading applications, starting applications, user interface preparation, service and component initialization, and the specific content can be referred to above. If the voice interaction module is already running in the background, but the voice interaction module is in a dormant or low-power state, the AP wakes up the voice interaction module and restores the voice interaction module to a normal working state, and the specific content can be referred to above. If some functions of the voice interaction module are running in the background, the AP restores the complete functions of the voice interaction module, including but not limited to: user interface display and background services.

[0078] It should be understood that the above is an example of an application scenario and does not limit the application scenario of the present application.

[0079] It can be understood that the visual interface elements such as text and controls displayed in the above interface are only examples, and this application does not limit the display position, display style, display size and other parameters of the interface.

[0080] In the above-mentioned breath awakening process, when the electronic device is in the screen-off state, the AP is in sleep mode, and the ADSP works in low-power mode. It needs to continuously perform gesture detection, and after detecting the hand-raising gesture, it switches to the normal working mode to collect audio signals, and then determines whether it is necessary to trigger the breath awakening, so as to decide whether to wake up the AP. After waking up the AP, the ADSP also needs to cache the audio signal and forward it to the AP so that the AP can perform subsequent voice recognition and voice interaction responses. ADSP has low-power space and non-low-power space. Due to the limited space capacity of the low-power space, the above-mentioned breath awakening detection and triggering will also subdivide each step to determine whether it should be executed in the low-power space or in the non-low-power space. In particular, steps such as breath awakening detection and audio data access that require a large memory space need to be arranged in the non-low-power space as much as possible, while gesture detection and breath awakening can be arranged in the low-power space. Therefore, the actual deployment scenario is that the electronic device is in the screen-off state, and the ADSP continuously performs gesture detection in the low-power working mode. When the hand-raising gesture is detected, it is necessary to switch to the normal working mode and continue for a period of time (for example, 2 seconds, 3 seconds, etc.) to collect voice signals, and it is also necessary to determine whether to trigger the breath wake-up function based on the collected voice signals. When it is determined that the breath wake-up function is to be triggered, the subsequent steps are executed. This brings new problems. Because there are many situations in which the posture of the electronic device changes in the screen-off state, it may cause the ADSP to switch to the normal working mode for a period of time each time the posture changes to try to collect voice signals of sufficient length, and to reserve enough pause time for the user before inputting voice, such as the above-mentioned fixed length of 2 seconds, 3 seconds, etc., and during the collection of voice signals, the ADSP is in the normal working mode, and the modules of the non-low-power space and the low-power space are working, which brings additional power consumption. The above-mentioned non-breath wake-up posture changes may include, for example, that the user may not speak after lifting the electronic device (for example, just blowing air on the screen to wipe the screen), or that the user may just flip the electronic device, or that the user may pick up the electronic device and then quickly put it down. Other situations are not listed one by one. When the above action occurs, the ADSP of the electronic device will start the breath wake-up detection process because of detecting the hand-raising gesture, and start waiting for a period of time, and receive a voice signal at any time point during this waiting time and start to continuously collect voice signals until the user stops inputting, and judge whether to trigger the breath wake-up function based on the collected voice signal. This results in that, when triggered by the above-mentioned breath wake-up posture change, the ADSP switches to the normal working mode and wastes this part of the power consumption.

[0081] In response to the above problems, this application provides a new voice interaction method. When the voice signal is collected after the hand-raising gesture is detected, the real-time gesture detection results will continue to be read, and whether to terminate the breath awakening will be determined based on the new gesture detection results. This solution mainly determines whether to continue the detection process after the breath awakening detection process has been triggered by combining the real-time gesture detection results, so that the breath awakening will be directly terminated when a gesture that does not meet the breath awakening occurs, without collecting audio signals and subsequent judgment. In the traditional scheme, after detecting the hand-raising gesture, the ADSP will be awakened to work in the normal working mode (non-low-power working mode) and continue for a fixed period of time (a preset fixed period of time, such as a fixed 2 seconds or 3 seconds, etc.), so as to reserve enough pause time for the user to input voice, and once the audio signal is collected within the fixed period of time, it will try to continue to collect the audio signal, and make subsequent judgments and wake up the AP based on the collected audio signal. This fixed period of time and the steps of collecting and subsequently judging the audio signal will continue to be executed even in the above-mentioned non-breath wake-up gesture, thereby causing a waste of power consumption. In response to this problem, the present application will continue to obtain real-time gesture detection results when starting to collect voice signals, so that the unnecessary "empty running" process can be terminated in time based on the new gesture detection results. In other words, if the above-mentioned non-breath wake-up gesture is encountered again, there is no need to wait for the fixed-time countdown to end, and it can be terminated in advance, thereby further reducing power consumption.

[0082] In order to facilitate the understanding of the present application scheme and the traditional scheme, the following is an explanation with specific examples. Scenario 1: Assume that the preset fixed duration is 2 seconds, and the electronic device is a mobile phone, which is placed on a table with the screen off. At this time, the ADSP of the mobile phone works in low power consumption mode and continuously performs gesture detection. Suppose a user picks up a mobile phone on the table and puts it into his pocket while chatting with someone. Then, during the picking process, the gesture detection module in the low-power space of ADSP can detect the hand-raising gesture. The mobile phone responds to the hand-raising gesture, wakes up ADSP and switches it to normal working mode, and maintains the normal working mode for at least 2 seconds. The breath wake-up detection module in the non-low-power space will collect audio signals. In the above scenario, the mobile phone is bound to start collecting the user's audio signal within the 2-second waiting time. Suppose the audio signal starts to be collected 1 second after the ADSP is awakened, and the collection is completed 4 seconds after being awakened. Then, in addition to the power consumption of the low-power space for more than 4 seconds, the ADSP also increases the power consumption within these 4 seconds, including the power consumption of the 1-second waiting time and the 3-second power consumption of collecting audio signals, as well as the computing power consumption when executing the subsequent steps of determining whether the audio signal can be used to wake up the AP. Due to the above scenario, users will not put the phone microphone close to their mouths, so the 3-second audio signal collected may be ordinary chatting sound, which does not meet the energy intensity requirement of the audio signal awakened by breath (the maximum intensity of the audio signal collected by the two microphones is still insufficient, or the intensity difference of the audio signals of the two microphones on standby is insufficient), and thus the AP cannot be awakened. Or, even if the AP can be awakened, the input voice signal does not involve user intent (such as Q&A or opening an app) after being recognized, which results in invalid processes after the complete audio signal is collected.If the present application scheme is adopted, in the above scenario, assuming that the user picks up the mobile phone on the table and kicks it into his pocket while chatting with someone, then, during the picking process, the gesture detection module of the low-power space of ADSP can detect the hand-raising gesture, and the mobile phone responds to the hand-raising gesture, wakes up ADSP to make it switch to normal working mode, and maintains the normal working mode for at least 2 seconds, the breath wake-up detection module of the non-low-power space will collect audio signals, and the breath wake-up detection module of the non-low-power space will continue to obtain real-time gesture detection results. In the above scenario, assuming that the new real-time gesture detection result is a non-lifting gesture at 0.8 seconds after ADSP is awakened (kicking the pocket causes the posture of the mobile phone to change again, and it is no longer in the lifted state), then, even if the voice signal is collected from the first second mentioned above, ADSP terminates the breath wake-up based on the new gesture detection result at 0.8 seconds, and no longer needs to execute subsequent processes such as collecting audio signals. The actual power consumption includes the continuous power consumption of the ADSP low-power space for 0.8 seconds, and the power consumption of the non-low-power space waiting for the collection of voice signals during this 0.8 seconds. It can be seen that the power consumption is much lower than that of the traditional solution in the same situation. Scenario 2: Assume that the preset fixed duration is 2s, the electronic device is a mobile phone, and the phone is in the off-screen state. At this time, the ADSP of the mobile phone is working in low-power mode and continuously detecting gestures. Assume that the user flips the phone and the user does not make any sound during the whole process, and the phone is not close to the mouth, and the user flips the phone in an instant and quick action. Assume that the entire flipping action lasts for 0.2 seconds, the first 0.06 seconds is to lift the phone, and the last 0.14 seconds is to turn the phone over. Then, in the process of lifting the phone before turning it over, the gesture detection module in the low-power space of ADSP can detect the hand-raising gesture. In response to the hand-raising gesture, the mobile phone wakes up the ADSP to switch it to normal working mode and maintains the normal working mode for at least 2 seconds. The breath wake-up detection module in the non-low-power space will collect audio signals. In the above scenario In this case, since flipping the phone is a quick action that occurs in an instant and the user does not make any sound, ADSP will not collect any audio signals within the 2 seconds. After the 2 seconds, ADSP stops the breath wake-up because no audio signals are collected, and switches back to low-power working mode, performing only gesture detection. During this process, the power consumption of ADSP includes not only the power consumption for 2.06 seconds (gesture detection in the low-power space during the 2-second normal working mode and gesture detection in the low-power space in the low-power mode for 0.06 seconds), but also the power consumption for trying to collect audio signals in the non-low-power space in the normal working mode during the 2-second hold.In the same situation, if the solution of the present application is adopted, the ADSP is woken up after the mobile phone detects the lift gesture at 0.06 seconds, but continues to receive the gesture detection results. Then, after 0.14 seconds in the normal working mode of ADSP, the breath wake-up is directly terminated because a new gesture detection result (no longer a lift gesture) is received, and there is no need to wait for the 2-second countdown. In this process, the power consumption of ADSP includes not only the power consumption of the low-power space of about 0.2 seconds, but also the power consumption of the non-low-power space of about 0.14 seconds. It can be seen that it is much lower than the power consumption when the traditional solution is adopted in the same scenario.

[0083] The above examples are only to illustrate how the present application scheme can effectively reduce power consumption. There is no limitation on the specific values, but it is only to make a more intuitive comparison of the reduction in power consumption. Other situations will not be listed one by one. In short, the present application scheme continues to receive gesture detection results in the normal working mode of ADSP, and determines whether to terminate the breath wake-up based on the real-time gesture detection results, thereby eliminating the invalid "empty run" process caused by the hand-raising gesture that is not awakened by the breath, so as to effectively reduce power consumption.

[0084] Figure 4 Schematic diagram of the software architecture of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device includes an application processor (AP) and an audio digital signal processor (ADSP). The AP may be an Android system, which may include an application layer (APP layer), a framework layer (framework, FWK), and a hardware abstraction layer (hardware abstraction layer, HAL), etc.

[0085] The application layer may include an application (APP). The application layer includes all applications installed on the electronic device, which may include voice interaction applications, such as smart assistants. Smart assistants are used to provide voice interaction functions. Smart assistants can parse the user's voice instructions and perform related operations according to the user's voice instructions, thereby realizing voice interaction between the electronic device and the user. The framework layer provides an application programming interface (API) and a programming framework for the application of the application layer. The framework layer includes some predefined functions. The framework layer may include an audio trigger module (ST), an audio policy service module, an audio recording module, etc. The audio trigger module of the framework layer is used to control the start of the voice interaction application of the application layer. The audio policy service module is used to establish a voice recognition channel with the voice interaction application. The audio recording module is used for audio recording. In some embodiments, the audio recording module can be implemented as AudioRecord, which is an API class for audio recording provided by the Android framework layer. The framework layer may also include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. The hardware abstraction layer is a layer between hardware and software. The hardware abstraction layer may include an audio trigger module, etc. The audio trigger module of the hardware abstraction layer (HAL layer) may also be referred to as an audio trigger driver (sound trigger hal, st-hal), and the hardware abstraction layer may also include an audio trigger module that may include an audio trigger pre-driver (primary-hal). The audio trigger module of the hardware abstraction layer is responsible for the audio trigger function, which enables applications to listen to certain sound events, such as the start of breath wake-up recognition notifications, in a low-power and privacy-sensitive manner. The audio trigger module of the hardware abstraction layer is used to process the hardware recognition of the start of breath wake-up recognition notifications and other sounds. Multiple drivers for driving the hardware operation may be installed in the hardware abstraction layer. It should also be understood that the above-mentioned audio trigger driver (st-hal) and audio trigger pre-driver (primary-hal) can also be referred to as an audio trigger module as a whole, which is used to trigger the audio function.

[0086] It should be noted that the application layer, framework layer and hardware abstraction layer may also include other contents, which are not specifically limited here. The embodiment of the present application is only illustrated by taking the Android system as an example. In other operating systems (such as Windows system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those of the embodiment of the present application, the solution of the present application can also be implemented.

[0087] like Figure 4As shown, the AP communicates and / or is electrically connected to the ADSP, and the ADSP communicates and / or is electrically connected to hardware devices such as microphones and sensors. The microphone may include a first microphone and a second microphone, and the first microphone and the second microphone may be respectively arranged at different positions of the electronic device, such as the top and the bottom. The sensor may include at least one of an acceleration sensor, a gyroscope, and an inertial detection unit, etc., for gesture detection.

[0088] ADSP can include low-power space (low-power memory space) and non-low-power space (non-low-power memory space). Low-power memory space and non-low-power memory space can be understood as different memory areas divided according to power optimization requirements. Low-power memory space uses power optimization technology to reduce power consumption, while non-low-power memory space focuses more on performance improvement, but power consumption may be higher. ADSP's low-power memory space usually refers to a memory area designed to run in ADSP low-power mode, which can remain active when some parts of the processor enter a low-power state. The design of ADSP's low-power memory space allows specific tasks to continue to be processed when other parts of the system (such as AP) are dormant, such as processing audio data or sensor data, thereby reducing overall power consumption. ADSP's non-low-power memory space can be a normal memory area, which provides data storage and processing functions when the system is active, but may not remain active when ADSP enters low-power mode. ADSP's non-low-power memory space can be used for services that need to continue running after the AP (application processor) is dormant, such as voice wake-up, sensor data processing, step counting, etc. ADSP's low-power memory space can be the secondary cache of a general-purpose processor (CPU). The secondary cache can provide fast data access, and the secondary cache can be implemented by low-power memory to reduce power consumption. The low-power memory space of ADSP can be implemented by low-power memory. Low-power memory has low power consumption characteristics and is suitable for specific services that need to continue to run when the AP is dormant. Low-power memory can include but is not limited to: static random-access memory (SRAM). As long as the SRAM remains powered on, the data stored in it can be kept constant. SRAM does not require a refresh circuit to save its internally stored data, which makes it relatively low power consumption during operation. Due to the static storage characteristics of SRAM, it consumes less power when not performing read and write operations, which is very suitable for low-power memory space with strict power consumption requirements. In the low-power memory space of ADSP, the use of SRAM can ensure stable storage of data while reducing overall power consumption.

[0089] like Figure 4As shown in the figure, the low-power memory space of ADSP includes the audio area and the sensor area. The audio area and the sensor area can be understood as different working areas divided according to business needs. The audio area is mainly responsible for audio-related businesses, such as processing tasks directly related to audio signals. The audio area can run algorithms related to audio processing. The sensor area is mainly responsible for tasks related to sensor data, such as the collection and processing of sensor data. The sensor area can run algorithms related to sensor data processing.

[0090] In an embodiment of the present application, the low-power memory space of ADSP is divided into different physical areas, and the sensor space (i.e., the sensing area) and the audio space (i.e., the audio area) are separated. The sensing area can be implemented by a sensor low power island (slpi), and the audio area can be implemented by audio low power island (audio low power island) technology. Both the sensor low power island and the audio low power island can be implemented by using low power memory such as SRAM as a memory.

[0091] In ADSP, SLPI can be understood as a hardware architecture specially designed for processing sensor data in low-power mode. The SLPI architecture is used to implement low-power memory space related to sensor data, such as the sensing area. Specifically, it can include: by integrating the drivers and algorithms of specific services (such as sensors) and other modules that need to run continuously into the SLPI space, so that they can be powered by SLPI alone and continue to run when the AP is asleep. In this way, the AP can enter a deep sleep state to reduce power consumption, while the SLPI space is responsible for processing tasks that need to run continuously. As a result, the AP can enter a sleep state when it is not needed, thereby reducing the power consumption of the overall system.

[0092] In the slpi architecture, sensors (such as IMU, gyroscope sensors and accelerometers, etc.), microphones, etc. can be electrically connected to ADSP so that ADSP can process the data collected by the sensors. When the AP is in sleep mode to reduce power consumption, ADSP can process the sensor data, thereby achieving lower power consumption while ensuring that some functions are not affected by AP sleep mode.

[0093] Exemplarily, the above-mentioned sensors and microphones are mounted on the ADSP, such as the above-mentioned sensors and microphones are integrated or connected to the same circuit board where the ADSP is located. In some embodiments, the drivers of the sensors and microphones run on the ADSP. In other embodiments, the drivers of the sensors and microphones run on the application processor (AP). The AP interacts with the sensors and microphones through a communication interface (such as I2C, SPI, UART, etc.) with the ADSP.

[0094] In ADSP, the audio low-power island is a functional area designed specifically for audio processing, used to process audio data in low-power mode. It can continue to run when the main processor is dormant, thereby reducing the power consumption of the entire system. This design allows audio data to be processed without CPU intervention. For example, in scenarios such as waking up the voice interaction module, voice data can be directly passed without waking up the CPU.

[0095] The audio low-power island usually takes advantage of the hardware characteristics of the system, such as using low-power memory SRAM as storage, because SRAM does not need to be refreshed as frequently as dynamic random access memory (DRAM), so the power consumption is low. This space can be used to store business data that needs to continue to run after the AP is dormant, such as sensor data processing, step counting, voice wake-up, etc.

[0096] In an embodiment of the present application, by placing some modules in the breath wake-up module that need to run continuously in the low-power memory space of the ADSP, it can be ensured that these modules can still operate normally when the AP is in sleep mode without significantly affecting the overall power consumption. The division of the low-power memory space of the ADSP allows audio and sensor data processing to be processed by a dedicated DSP (such as the ADSP) when the main processor is not active, thereby ensuring lower power consumption. In this way, the ADSP can continue to provide high-quality audio processing capabilities while maintaining low power consumption.

[0097] Optionally, some modules in the breath awakening module that need to run continuously are placed in the low-power memory space of ADSP, that is, some codes in the breath awakening module that need to run continuously are placed in the low-power memory space of ADSP.

[0098] It can be understood that the above-mentioned second memory is a low-power memory SRAM as an example. The functions of the low-power memory SRAM can all be implemented by the second memory. The embodiment of the present application only takes the low-power memory SRAM as an example, and does not mean that the present application is limited to this.

[0099] In the embodiment of the present application, the non-low-power memory space of the ADSP can be implemented by a non-low-power memory. The non-low-power memory may include, but is not limited to, DRAM, double data rate synchronous dynamic random access memory (DDR SDRAM), or other types of memory.

[0100] DRAM is a dynamic random access memory, and the data stored in it needs to be refreshed periodically to maintain data integrity. In the non-low-power memory space of ADSP, since power consumption is not the primary consideration, highly integrated and low-cost DRAM can be used as memory. The large capacity and relatively low power consumption of DRAM make it an ideal choice for non-low-power memory space storage.

[0101] DDR is a synchronous dynamic random access memory with double data rate. It uses double data rate transmission technology to transmit data at both the rising and falling edges of each clock cycle, thus achieving high-speed data transmission. DDR has a large storage capacity and supports a variety of capacity specifications to meet the needs of ADSP for large amounts of data storage. DDR can provide large-capacity, high-speed and low-power data storage and transmission support for non-low-power memory space.

[0102] It is understood that the low-power memory space and non-low-power memory space in ADSP can also be implemented by other memories. For example, the low-power memory space can also be implemented by low-power double data rate synchronous dynamic random access memory (LPDDRSDRAM). According to the specific requirements and power consumption constraints of ADSP, an appropriate memory type can be selected to implement the functions of low-power memory space and non-low-power memory space, and this application does not make specific limitations on this.

[0103] In the embodiment of the present application, the low power mode is a working mode designed by ADSP to reduce power consumption. When ADSP is in low power mode, in order to reduce overall power consumption, it will choose to use the low power memory space of ADSP. In non-low power mode, it will choose to use the non-low power memory space of ADSP, thereby obtaining the best performance and bandwidth.

[0104] Specifically, when executing the voice interaction method provided in the embodiment of the present application, when ADSP is in low-power mode, the low-power memory space of ADSP can be used, and the low-power memory space of ADSP is not used, that is, the low-power memory of the low-power memory space is in a working state, and the non-low-power memory of the non-low-power memory space is in a low-power state. When executing the voice interaction method provided in the embodiment of the present application, when ADSP is in a non-low-power mode, the non-low-power memory space of ADSP can be used, and the low-power memory space of ADSP can continue to be used, that is, the related modules of the non-low-power memory space and the low-power memory space are all in a working state.

[0105] In the embodiment of the present application, ADSP exiting low power mode and entering low power mode can be understood as switching the working mode of non-low power memory (such as DDR) in non-low power memory space. Taking non-low power memory DDR as an example, ADSP enters low power mode and DDR is in low power state. When DDR is in low power state, power consumption can be reduced by reducing the operating frequency, entering low power mode, etc. ADSP exits low power mode and DDR enters working state. When DDR is in working state, DDR may run at full speed or work according to the conventional memory access mode to meet ADSP's requirements for high performance and response speed.

[0106] In some embodiments, when DDR is in a low power state, the operating current of DDR is very small, and DDR may only need to maintain data without loss, but cannot read or write. When DDR is in a normal state, the operating current of DDR is very large. The operating current of DDR in a low power state is less than the operating current of DDR in a normal state.

[0107] In some embodiments, when DDR is in a low power state, the operating voltage of DDR is very small, and DDR may only need to maintain data without loss, but cannot read or write. When DDR is in a normal state, the operating voltage of DDR is very large. The operating voltage of DDR in a low power state is lower than the operating voltage of DDR in a normal state.

[0108] It should be understood that the present application only uses DDR and DRAM as examples to illustrate the functions of non-low-power memory, but other non-low-power memory may also be used in practice without limitation.

[0109] In the above text, ADSP is used as an example of a non-main processor (other processor), which includes a low-power space and a non-low-power space. If the present application scheme is extended to other similar scenarios, ADSP can also be replaced with other non-main processors. For example, when you want to make similar improvements in the field of image processing, you can use sensorhub processors, etc. However, when setting what functional modules and how to deploy them to low-power space and non-low-power space respectively, it will be different due to the actual application scenario, so it will not be repeated, and other situations will not be listed one by one.

[0110] Optionally, the low-power space of ADSP can be a low-power memory space, and the non-low-power space of ADSP can be a non-low-power memory space. The low-power memory space and non-low-power memory space of ADSP can be understood as different memory areas divided according to power optimization requirements. The low-power memory space uses power optimization technology to reduce power consumption, while the non-low-power memory space focuses more on performance improvement, but the power consumption may be higher. The relevant contents of the low-power space, non-low-power space, low-power memory space and non-low-power memory space of ADSP can be referred to above and will not be repeated here.

[0111] For example, the low power space or low power memory space of ADSP can select low power storage technology, such as low power memory, such as low power version of static random access memory (SRAM), or flash memory (flash) technology with low power characteristics. The relevant content of low power memory can be referred to above, and will not be repeated here.

[0112] Exemplarily, the non-low-power space or non-low-power memory space of ADSP can be implemented by a non-low-power memory, such as DDR or DRAM, etc. The relevant contents of the non-low-power memory can be referred to above and will not be repeated here.

[0113] In the traditional breath wake-up solution that divides the steps of breath wake-up and deploys them in different power consumption spaces, by placing the modules that need to run continuously in the breath wake-up module in the low-power memory space, it can be ensured that these modules can still run normally when the AP is dormant without significantly affecting the overall power consumption; by placing the remaining modules in the breath wake-up module in the non-low-power memory space, it can provide higher performance and bandwidth. High-performance and bandwidth memory can process data faster, making ADSP faster when processing large tasks, running complex programs or performing multiple tasks. The speed increase can significantly improve the efficiency of waking up the smart assistant, thereby improving the user experience.

[0114] Figure 4 This is an example of deploying different functional modules of the breath awakening module in the ADSP in the above traditional solution. Figure 4 As shown, the breath wake-up module of the electronic device includes a gesture detection module and a breath wake-up detection module. The gesture detection module is deployed in the low-power memory space of the ADSP, and the breath wake-up detection module is deployed in the non-low-power memory space of the ADSP. The gesture detection module is used to detect the motion state of the electronic device. Specifically, the ADSP runs the gesture detection module, obtains the motion data of the electronic device, and determines the motion state of the electronic device (that is, the posture change) according to the motion data of the electronic device.

[0115] The motion data may include, but is not limited to, IMU data. For example, the motion data may include acceleration data in the x-axis direction, acceleration data in the y-axis direction, and acceleration data in the z-axis direction acquired by an acceleration sensor of the electronic device. For another example, the motion data may include angular velocity data. Other cases are not listed one by one.

[0116] The gesture detection module may include a posture detection model. Taking the motion data as acceleration data of the electronic device in the x-axis direction, acceleration data in the y-axis direction, and acceleration data in the z-axis direction as an example, since the absolute values ​​of the acceleration data corresponding to the three coordinate axes of the x-axis, y-axis, and z-axis, d1, d2, and d3 can represent whether the electronic device is in motion, p1, p2, and p3 can represent the amplitude of the motion of the electronic device, and the difference between d1 and p1, the difference between d2 and p2, and the difference between d3 and p3 can represent the motion state of the electronic device from other dimensions such as the smoothness of the motion. Therefore, the posture detection model can comprehensively judge whether the electronic device is lifted up from multiple aspects such as whether the electronic device is moving, the amplitude of the motion, and the smoothness of the motion through the above-mentioned motion data, that is, judge whether the electronic device is in a lifted state or a lifting event occurs, thereby improving the accuracy of the posture detection model judgment.

[0117] The posture detection model can be a trained convolutional neural network model, a deep learning model, or any other neural network model that can perform posture detection. It can be obtained by training the neural network model with sufficient labeled training data. The training data may include multiple sets of motion data and posture labels corresponding to each set of motion data. It should also be noted that the present application is aimed at the detection of hand-raising gestures, so when training the posture detection model, only the detection of hand-raising gestures can be trained, and the postures of other non-hand-raising gestures are marked as "non-hand-raising gestures" to further reduce the requirements for the position detection of the model, that is, the posture labels in the training data only include "hand-raising gestures" and "non-hand-raising gestures". After training with such training data (samples), the posture detection model will only recognize hand-raising gestures and non-hand-raising gestures, without the need to further subdivide non-hand-raising gestures into flip gestures, rotation gestures, and other gestures. This further reduces the model size and improves the accuracy of detection to a certain extent.

[0118] Since the posture detection model of the present application is mainly used for gesture detection and is to be deployed in the low-power memory space of ADSP, a lightweight neural network model can be selected to train the posture detection model and deploy it to the low-power memory space of ADSP.

[0119] The gesture detection module will continue to detect gestures uninterruptedly when the screen is off. When a hand-raising gesture is detected, the breath wake-up detection module will be triggered for subsequent detection.

[0120] The breath awakening detection module is used to perform breath awakening detection.

[0121] In the embodiment of the present application, the gesture detection module detects that the electronic device is in the lifted state, and notifies the breath awakening detection module to obtain motion data and / or voice data. The breath awakening detection module determines whether to trigger the breath awakening function based on the obtained motion data and / or voice data.

[0122] In an embodiment of the present application, the breath awakening detection module may include a preliminary detection model, a voice detection model, a state detection model, and a voice-state detection model. The preliminary detection model is used to determine whether to detect whether the audio data received by the electronic device is detected. The voice detection model is used to determine whether the audio data is a voice command sent by the user to the electronic device. The state detection model is used to determine whether the electronic device is in a raised state to receive the audio data. The voice-state detection model is used to detect that the audio data received by the electronic device is a voice command and the electronic device is currently in a raised state.

[0123] The breath wake-up detection module can also wake up the voice interaction module according to other wake-up strategies, without limitation.

[0124] like Figure 4 As shown, the breath awakening module of the electronic device may further include a breath awakening starting module.

[0125] The breath awakening activation module is used to activate the breath awakening detection module when the motion state of the electronic device is a lifted state.

[0126] The implementation logic of the breath awakening start module can be a loop execution detection. For example, the loop execution detects whether the motion state of the electronic device is detected to be the lifted state, until the motion state of the electronic device is detected to be the lifted state. The breath awakening start module can be deployed in the low-power memory space of ADSP, so that it can quickly respond to the motion state of the electronic device being the lifted state.

[0127] The gesture detection module can be deployed in the sensing area of ​​the low-power memory space. The breath wake-up start module can be deployed in the audio area of ​​the low-power memory space. The gesture detection module deployed in the sensing area can continue to process sensor data when the AP is in sleep mode, such as determining the motion state of the electronic device by detecting motion data collected by the sensor. The breath wake-up start module deployed in the audio area can start other modules of the breath wake-up module (such as the breath wake-up detection module) in response to the motion state of the electronic device being the lifted state when the AP is in sleep mode, and then perform breath wake-up detection.

[0128] The breath awakening module of the electronic device may further include a state forwarding module. When the gesture detection module detects that the electronic device is in a lifted state, the lifted state may be transmitted to the breath awakening start module through the state forwarding module.

[0129] The state forwarding module can be implemented as a message transmission channel between the audio area and the sensor area. The message transmission channel is used to implement communication between processes or systems. Exemplarily, when the breath wake-up module is deployed on ADSP, the state forwarding module can be implemented through message transmission channels such as qsocket and Qualcomm messaging interface (QMI). When the breath wake-up module is deployed on other types of second processors, the state forwarding module can also be implemented through shared memory, etc. It can be understood that the present application does not specifically limit the specific implementation of the state forwarding module deployed on the second processor.

[0130] The state forwarding module can be deployed in the non-low-power memory space of ADSP. The breath awakening module of the electronic device can also include an audio data access module. The audio data access module is used to store the voice collected by the electronic device, so that after the voice interaction module is awakened later, the voice in the audio data access module is transmitted to the voice interaction module for voice recognition.

[0131] When the gesture detection module detects that the electronic device is in the lifted state, the breath awakening detection module is notified to obtain motion data and / or voice, and the audio data access module is notified to obtain voice. That is, after detecting that the electronic device is in the lifted state, the voice signals collected by multiple microphones are obtained, and the obtained voice signals are copied into two copies, one of which is transmitted to the breath awakening detection module, and the other is transmitted to the audio data access module.

[0132] Optionally, the audio data access module may be implemented as data access management (DAM) of the audio data.

[0133] The audio data access module can be deployed in the non-low-power memory space of ADSP. The audio data access module can be used as a module independent of the breath awakening module.

[0134] The breath awakening module of the electronic device may further include a breath awakening pre-module. The breath awakening pre-module is responsible for the pre-work of breath awakening detection. When the breath awakening start module detects that the electronic device is in a lifted state, the breath awakening start module starts the breath awakening pre-module.

[0135] When the breath awakening start module detects that the electronic device is in the lifted state, the breath awakening start module starts the breath awakening front module, and the breath awakening front module controls the ADSP to exit the low power consumption mode, obtains the motion data collected by the sensor, notifies the audio data access module to cache the audio data, and starts the breath awakening detection module.

[0136] The breath awakening front module may include an exit low power consumption unit, a motion data acquisition unit, a notification unit and a startup unit. The exit low power consumption unit is used to control the ADSP to exit the low power consumption mode. The motion data acquisition unit is used to acquire the motion data collected by the sensor. The notification unit is used to notify the audio data access module to cache the audio data. The notification unit can notify the audio data access module to cache the data by modifying the flag bit of the audio data access module. The startup unit is used to start the breath awakening detection module.

[0137] The breath awakening pre-module can be deployed in the low-power memory space and non-low-power memory space of ADSP. For example, the exit low-power unit is deployed in the low-power memory space of ADSP. For another example, the exit low-power unit is deployed in the audio area of ​​the low-power memory space. For another example, the motion data acquisition unit, notification unit and startup unit can be deployed in the non-low-power memory space of ADSP.

[0138] The breath awakening module may also include a detection processing module (not shown in Figure 4 ). The detection processing module is used to perform corresponding operations according to the detection results of the breath awakening detection module. When the breath awakening detection module detects that the breath awakening is successful, that is, it is determined that the breath awakening event is triggered, the operation corresponding to the successful detection is performed, such as waking up the AP and clearing the data and flag bits used for the current breath awakening detection. When the breath awakening detection module detects that the breath awakening fails, that is, it is determined that the breath awakening event is not triggered, the operation corresponding to the detection failure is performed, such as clearing the data and flag bits used for the current breath awakening detection and controlling the ADSP to enter low power consumption mode.

[0139] Optionally, when the breath awakening detection module detects a breath awakening event, it wakes up the AP and modifies a flag bit of the audio data access module to control the audio data access module to close the audio data cache.

[0140] Optionally, when the breath wake-up detection module detects that no breath wake-up event is triggered, it controls the ADSP to enter a low power consumption mode, and modifies a flag bit of the audio data access module to control the audio data access module to close the audio data cache.

[0141] The detection processing module can control ADSP to enter low power mode before performing breath awakening detection. The detection processing module can control the audio data access module to close the audio data cache before performing breath awakening detection. The detection processing module can be independent of the breath awakening detection module, or it can be used as a breath awakening detection module or as part of other modules. The detection processing module can be deployed in the non-low power memory space of ADSP.

[0142] When the gesture detection module detects that the electronic device is in the lifted state, the state forwarding module, the breath wake-up start-up module and the breath wake-up front module can be used to notify the breath wake-up detection module to obtain motion data and voice (such as the first voice and / or the second voice), and notify the audio data access module to obtain voice (such as the first voice and / or the second voice). In an embodiment of the present application, the various modules of the breath wake-up module are distributedly deployed in the ADSP. Specifically, the gesture detection module and the breath wake-up start-up module are deployed in the low-power memory space of the ADSP, while the state forwarding module, the breath wake-up front module, the breath wake-up detection module and the audio data access module are deployed in the non-low-power memory space of the ADSP, thereby reducing the dependence of the breath wake-up module on the low-power memory space of the ADSP.

[0143] The complex programs implemented by the gesture detection module and the breath wake-up startup module are not complicated and occupy less low-power memory space. Even if the low-power memory space of ADSP is limited, the gesture detection module and the breath wake-up startup module can be deployed in the low-power memory space of ADSP.

[0144] The gesture detection module and the breath wake-up start module are deployed in the low-power memory space of ADSP. Therefore, the gesture detection module and the breath wake-up start module can detect the movement state of the electronic device in real time when the AP is in sleep state, ensure real-time processing of the wake-up voice interaction module, and meet the high real-time requirements of wake-up.

[0145] The breath awakening detection module is deployed in the non-low-power memory space of ADSP. The space of the non-low-power memory space is sufficient to accommodate the breath awakening detection module. As well as the higher performance and bandwidth provided by the non-low-power memory space of ADSP, the detection efficiency of the breath awakening detection module can be improved.

[0146] The low-power exit unit of the breath awakening front module is deployed in the low-power memory space of ADSP, so that when the breath awakening startup module detects the lifting state, ADSP can be quickly controlled to exit the low-power mode.

[0147] The state forwarding module is deployed in the non-low-power memory space of ADSP to quickly forward the hand-raising state based on the higher performance and bandwidth provided by the non-low-power memory space of ADSP.

[0148] The motion data acquisition unit, notification unit and startup unit of the breath awakening front-end module are deployed in the non-low-power memory space of ADSP, so as to quickly trigger the breath awakening detection process based on the higher performance and bandwidth provided by the non-low-power memory space of ADSP.

[0149] The audio data access module is deployed in the non-low-power memory space of ADSP to improve data cache efficiency based on the higher performance and bandwidth provided by the non-low-power memory space of ADSP.

[0150] For example, the low-power memory of the low-power memory space is SRAM, the non-low-power memory of the non-low-power memory space is DRAM, and the motion data acquired by the gesture detection module is IMU data collected by the IMU.

[0151] It can be understood that, taking the first memory as a non-low-power memory DRAM as an example, the functions of the non-low-power memory DRAM can be realized through the first memory, and taking the second memory as a low-power memory SRAM as an example, the functions of the low-power memory SRAM can be realized through the second memory. The embodiments of the present application only take the non-low-power memory DRAM and the low-power memory as SRAM as an example for explanation, and it does not mean that the present application is limited to this.

[0152] Figure 5 It is a schematic diagram of the execution process of the breath awakening module during a voice interaction process in an embodiment of the present application. Figure 5 It mainly provides examples of the steps executed by each breath awakening-related module from the internal software architecture layers during voice interaction. Figure 5 It can also be seen as an explanation of the breath awakening process in the voice interaction solution of the present application from the perspective of the collaborative work of various internal modules. Figure 5 For more information about the application processor (AP) and audio digital signal processor (ADSP) and the modules in it, please refer to Figure 4 , no more details. Figure 5 Mainly take the example that the low-power memory of the low-power memory space of ADSP is SRAM, the non-low-power memory of the non-low-power memory space is DRAM, and the motion data obtained by the gesture detection module is IMU data collected by the IMU.

[0153] like Figure 5 As shown, the IMU of the electronic device collects IMU data in real time.

[0154] The gesture detection module obtains the IMU data collected by the IMU. When the gesture detection module detects a hand-raising event based on the IMU data, the hand-raising event is sent to the state forwarding module.

[0155] The state forwarding module determines that the electronic device is in the raised state in response to the hand raising event. Specifically, the state forwarding module can obtain the hand raising event through inter-process or inter-system communication, and then determine that the electronic device is in the raised state in response to the hand raising event. The state forwarding module sends the raised state to the breath wake-up start module.

[0156] The breath awakening start module responds to the electronic device being in the lifted state, triggering the breath awakening front module to start. When the breath awakening front module starts, it performs the following operations: 1. Control ADSP to exit low power mode; 2. Obtain IMU data and transmit IMU data to the breath awakening detection module; 3. Notify the audio data access module to cache audio data; 4. Start the breath awakening detection module to perform breath awakening detection.

[0157] After the breath awakening detection module is started, the breath awakening detection module obtains the audio data collected by the first microphone and the second microphone. The breath awakening detection module also obtains the IMU data transmitted by the breath awakening front-end module.

[0158] After the audio data access module is started, the audio data access module obtains the audio data collected by the first microphone and the second microphone and stores the obtained audio data; and, in response to the electronic device being in a raised state, the audio data collected by the first microphone and the second microphone are also transmitted to the breath wake-up detection module.

[0159] When the breath wake-up detection module detects a breath wake-up event (that is, determines that the breath wake-up function can be triggered) based on the obtained audio data and / or motion data (such as IMU data), it wakes up the application processor AP. The smart assistant is started or woken up through the AP.

[0160] After the breath awakening detection module detects a breath awakening event, the breath awakening detection module can also be used to control the audio data access module to close the data cache. After starting or waking up the smart assistant, the smart assistant can obtain the audio data cached by the audio data access module and perform voice recognition and corresponding operations.

[0161] like Figure 5As shown, the gesture detection module (module in SRAM) will only continuously detect and report the hand-raising event when the ADSP is in low-power working mode. Mainly, the IMU continuously reports the detected motion data, and then the gesture detection module determines whether a hand-raising event has occurred based on the IMU data. Once it is determined that a hand-raising event has occurred, the gesture detection module will inform the breath wake-up start module of the current lifting state after the lifting event is transmitted through the state forwarding module, and then the breath wake-up start module further reports the lifting state to the breath wake-up front module and the breath wake-up detection module (module in DRAM) to wake up the ADSP to work in the normal working mode and trigger the breath wake-up detection step. After that, the breath wake-up detection module determines whether to trigger a breath wake-up event (that is, whether to start the breath wake-up function) based on the voice signals collected from the two microphones and combined with the reported IMU data in the normal working mode of the ADSP. After determining that the breath wake-up function is to be performed, the AP is awakened, and the AP reads the cached audio data from the audio data access module of the ADSP (module in DRAM). These cached audio data are collected by two microphones and sent synchronously to the breath wake-up detection module and the audio data access module. The audio data used in the two modules is the same. After that, the AP will perform subsequent voice recognition and other voice interaction responses, which will not be repeated here.

[0162] However, due to the variety of user gesture operations on electronic devices, a hand-raising event may not be accompanied by breath wake-up. As a result, after the gesture detection module detects the hand-raising event, the user does not further input a voice signal, resulting in the ADSP being invalidly awakened, and even the AP being invalidly awakened. To address this problem, the present application allows the gesture detection module to continue to report hand-raising events, even after the ADSP is awakened, so that the breath wake-up start module can decide whether to trigger a breath wake-up event based on the real-time lifting status (lifted or not lifted), thereby effectively reducing power consumption. This method of determining whether to continue executing subsequent processes based on new posture changes can avoid the ADSP being mistakenly awakened by a lifting gesture (lifting event) that is not intended for breath wake-up.

[0163] Figure 6 is a schematic flow chart of a method for voice interaction according to an embodiment of the present application. Figure 6 Each step is described below.

[0164] S601. At a first moment, the electronic device is lifted up and the screen of the electronic device is in an off state, and the power consumption of the electronic device changes from a first value to a second value. Before the first moment, the screen of the electronic device is in an off state, the electronic device is not lifted up, and the second value is greater than the first value.

[0165] The first moment can be understood as the moment when the electronic device detects that it is lifted, for example Figure 3 The moment when the electronic device detects the hand being lifted during the time period of (b) in the figure. In some implementations, the first moment may be the moment when the electronic device detects the hand-raising event (hand-raising gesture). The moment before the first moment may be understood as the moment when the electronic device is not lifted in the off-screen state, for example Figure 3 Any moment in the time period of (a) can be considered as a moment when the electronic device is not lifted.

[0166] S602: At a second moment after the first moment, the electronic device is not lifted, and the screen of the electronic device is in an off state, and the power consumption of the electronic device changes from the second value to the first value.

[0167] That is to say, if the electronic device is still in the off state, and when the lifting state of the electronic device changes, the power consumption returns to the original lower value.

[0168] In one implementation, at a second moment after the first moment, the electronic device is not lifted, and the screen of the electronic device is in an off-screen state, and when the power consumption of the electronic device changes from the second value to the first value, it may include: at the second moment, based on the electronic device changing from a lifted state to a not lifted state, the power consumption of the electronic device changes from the second value to the first value. In this implementation, it is explained why the power consumption value changes at the second moment, because the electronic device changes based on the change in the lifted state in the off-screen state. It can also be understood that the power consumption change at the second moment occurs under the triggering of the change in the lifted state from lifted to not lifted. Combined with the above first moment, when the lifted state of the electronic device changes from not lifted to lifted, it constitutes a complete solution for changing the power consumption based on the change in the lifted state of the electronic device. In the traditional scheme, there will not be a second moment like the present application (or it can be understood that the second moment will not be like the present application where the electronic device will end the process based on the change in the lifting state). In the traditional scheme, once the electronic device is lifted and the power consumption changes from the first value to the second value, it is necessary to wait for at least the third moment below (no suitable voice data is collected within the preset time period) or the sixth moment (the entire voice interaction process is completed) before returning to the first value. Therefore, compared with this, the present application has the effect of "stopping losses in time", and can end the process in time after the lifting state changes, without wasting power consumption by executing subsequent processes in vain.

[0169] In another implementation, the method further includes: before the first moment, the first processor of the electronic device continuously detects hand-raising events in a low power consumption mode, and the hand-raising events are used to characterize whether the electronic device is lifted, wherein when the hand-raising event is detected, it indicates that the electronic device is lifted, and when the non-hand-raising event is detected, it indicates that the electronic device is not lifted; at the first moment, in response to detecting the hand-raising event, the first processor switches from the low power consumption mode to the normal working mode, and begins to determine whether a breath wake-up event is triggered in the normal working mode, and continues to obtain the detection results of the hand-raising event in the normal working mode; at the second moment, in response to detecting the non-hand-raising event, the first processor switches from the normal working mode to the low power consumption mode, and continuously detects hand-raising events in the low power consumption mode. In this implementation, the first processor continuously detects hand-raising events in low power consumption mode. Once a hand-raising event is detected, it indicates that the electronic device has been lifted up, that is, the first moment arrives. The electronic device switches from low power consumption to normal working mode in response to detecting the hand-raising event, so that the power consumption increases from the previous first value to the subsequent second value starting from the first moment. From the first moment to the second moment, the first processor is in normal working mode until the second moment when the electronic device detects a non-hand-raising event again, and thus knows that the lifting state has ended. Based on this, the first processor is directly returned to the low power consumption mode, instead of waiting for a sufficient period of time and executing subsequent processes, thereby effectively reducing power consumption.

[0170] The first processor may be an ADSP or other processor capable of deploying a breath awakening module and having a low-power memory space. The second processor may be an AP or other processor capable of performing speech recognition and responding to the recognition result.

[0171] In another implementation, the method further includes: between the first moment and the third moment, the electronic device remains lifted up, and the screen of the electronic device is in an off state, the time interval between the first moment and the second moment is less than the time interval between the first moment and the third moment, and the time interval between the first moment and the third moment is a preset time interval; at the third moment, the power consumption of the electronic device changes from the second value to the first value. In this implementation, the step of maintaining a fixed waiting time of the traditional solution is retained, so that the timely termination of the process can be ensured on the basis of superimposing the solution of the present application, that is, if the second moment does not occur, that is, the lifting state of the electronic device does not change, then the process ends after the preset time interval is counted from the first moment, and the power consumption is reduced back to the first value.

[0172] In one example, at the third moment, the power consumption of the electronic device changes from the second value to the first value, including: at the third moment, based on the electronic device not collecting voice data that meets the first preset condition, the power consumption of the electronic device changes from the second value to the first value, and the first preset condition includes: the signal strength of the voice data collected by the first microphone of the electronic device is greater than or equal to the first preset signal strength threshold, the signal strength of the voice data collected by the second microphone of the electronic device is greater than or equal to the second preset signal strength threshold, and the difference between the signal strength of the voice data collected by the first microphone and the signal strength of the voice data collected by the second microphone is greater than or equal to the preset signal strength difference threshold. In this example, the triggering factor for adjusting the power consumption at the third moment is given, that is, based on the fact that the electronic device does not collect voice data that meets the first preset condition. In combination with the above, there are two ways in which the present application can trigger "returning to low power consumption", the lifting state is changed (or understood as the electronic device ending the lifting state) and the voice data that meets the first preset condition is not collected within the preset time interval. The two triggering methods can be executed separately or superimposed. Only executing the first method is the present application solution, and only executing the second method is the traditional solution. The best effect of reducing power consumption is to execute both methods.

[0173] In one example, at the third moment, based on the fact that the electronic device does not collect voice data that meets the first preset condition, the power consumption of the electronic device changes from the second value to the first value, including: before the first moment, the first processor of the electronic device operates in a low power consumption mode; between the first moment and the third moment, the first processor operates in a normal working mode; the first processor continuously reads the voice data collected by the first microphone and the second microphone from the first moment; at the third moment, the first processor determines that the breath awakening event is not triggered based on the fact that the electronic device is lifted and the electronic device does not collect voice data that meets the first preset condition, controls the first processor to switch from the normal working mode to the low power consumption mode, and continuously detects the hand raising event in the low power consumption mode. In this example, it is mainly explained that the numerical change of power consumption is caused by the change of the working mode of the first processor. Before the first moment, the first processor is in a low power consumption mode, so the power consumption is the lowest first value, between the first moment and the third moment is a normal working mode, so the power consumption is a higher second value, at the third moment, because no voice data that meets the first preset condition is collected, the breath awakening event is not triggered, so that the first processor is put into the low power consumption mode again, so the whole process shows that the power consumption changes from the first value to the second value and then back to the third value.

[0174] It should be noted that in this example, the electronic device remains in the lifted state at the second moment (the electronic device is continuously lifted from the first moment to the third moment). Therefore, the process of ending the power consumption based on the change of the lifted state at the second moment in step S602 will not occur. Instead, the power consumption will be turned off based on the failure to collect suitable voice data at the third moment.

[0175] In one implementation, the method further includes: at a fifth moment after the first moment, the electronic device turns on the screen, and the power consumption of the electronic device is a third value, and the third value is greater than the second value. In this implementation, the power consumption change caused by the successful breath awakening is mainly given. The moment when the interface 107 just lights up can be regarded as an example of the third moment.

[0176] In one example, at a fifth moment after the first moment, the screen of the electronic device is turned on, and the power consumption of the electronic device is a third value, which is greater than the second value, including: at the first moment, the first processor of the electronic device is switched from a low power consumption mode to a normal working mode, thereby increasing the power consumption of the electronic device from the first value to the second value; at a fourth moment after the first moment, based on the electronic device starting to collect voice data that meets the first preset condition, the first processor starts to continuously read the collected voice data; the first preset condition includes: the signal strength of the voice data collected by the first microphone of the electronic device is greater than or equal to the first preset signal strength threshold, and the signal strength of the voice data collected by the second microphone of the electronic device is greater than or equal to the first preset signal strength threshold. The degree is greater than or equal to the second preset signal strength threshold, and the difference between the signal strength of the voice data collected by the first microphone and the signal strength of the voice data collected by the second microphone is greater than or equal to the preset signal strength difference threshold; at the fifth moment after the fourth moment, the first processor has finished reading the first voice data, and the first processor wakes up the second processor based on the first voice data satisfying the second preset condition, so that the second processor switches from the low power consumption mode to the normal working mode and performs voice recognition and response to the first voice data in the normal working mode, thereby increasing the power consumption of the electronic device from the second value to the third value, and satisfying the second preset condition is used to indicate that the first voice data can be used to trigger the breath awakening event. In this example, it is given how the power consumption changes from the first moment to the fifth moment change with the working mode of the processor. Before the first moment, the first processor and the second processor are both in low power consumption mode, between the first moment and the fourth moment, the first processor is in normal working mode, between the fourth moment and the fifth moment, the first processor is in normal working mode, and after the fifth moment, the second processor and the first processor are both in normal working mode, so that when the awakening is successful, the power consumption continues to increase.

[0177] In another example, the method further includes: at a fifth moment, based on the first voice data not satisfying the second preset condition, the first processor switches from the normal working mode to the low power consumption mode, so that the power consumption of the electronic device changes from the second value to the first value. In this example, when the first processor determines that the breath wake-up event is not triggered, it switches back to the low power consumption mode.

[0178] In another example, the method further includes: at a sixth moment after the fifth moment, the screen of the electronic device changes from the on state to the off state, and the power consumption of the electronic device changes from the third value to the first value. In this example, when the screen of the electronic device turns off again, the power consumption returns to the first value.

[0179] In another example, at a sixth moment after the fifth moment, the screen of the electronic device changes from the on state to the off state, and the power consumption of the electronic device changes from the third value to the first value, including: at the sixth moment, based on the second processor completing the response to the first voice data, the first processor and the second processor both switch to a low power consumption mode, so that the power consumption of the electronic device changes from the third value to the first value. In this example, after the second processor completes the response, both processors are switched to a low power consumption mode, thereby reducing the power consumption to the first value.

[0180] In another example, the method further includes: at the fourth moment, based on the electronic device not collecting voice data that meets the first preset condition, the first processor does not read the voice data that meets the first preset condition, and the electronic device continues to collect voice data until the third moment, and the time interval between the third moment and the first moment is the preset time interval. In this example, if the voice data has not been collected at the fourth moment, the voice data will continue to be collected until the third moment arrives.

[0181] It should also be noted that in addition to the first and second preset conditions, a third preset condition can be set before the first preset condition. The third preset condition can be identity authentication, that is, voiceprint recognition. Only voice data that meets the voiceprint recognition can execute the subsequent process. For example, Fig. 9 It gives an example of recording a conversation after voiceprint recognition. The so-called beginning of the conversation recording can be regarded as an example of the fourth moment in this article.

[0182] Figure 6 The method shown is mainly to reduce power consumption in time when the electronic device is raised and then stops being raised during the voice interaction process in the breath awakening scenario, thereby eliminating unnecessary power consumption caused by raising the hand for non-breath awakening, so as to reduce overall power consumption. Figure 6 The method shown can be Figure 1-Figure 2 After the setup process is completed, Figure 6 The method shown can be used for Figure 3 The scene shown, Figure 6 The method shown can be used Figure 4 , Figure 5 The execution of each module shown in the figure, the relevant content is referred to above and will not be repeated here.

[0183] The complete process of voice interaction in this application involves three stages: breath awakening detection stage, voice recognition stage and dialogue stage. The breath awakening detection stage mainly detects gestures and whether a breath awakening event is triggered. The voice recognition stage is mainly the stage of voice recognition after determining that a breath awakening event is to be triggered. The dialogue stage is the stage of collecting audio data in the breath awakening detection stage. Figure 3 , the breath awakening detection stage corresponds to Figure 3 (a)-(b) in the figure and the stage after the user inputs the voice; the voice recognition stage corresponds to the interface 107, and the text information corresponding to the voice data input by the user is displayed in the interface through voice recognition; the dialogue stage can correspond to Figure 3 (b) The user speaks. The breath wake-up detection phase is mainly performed by ADSP, while the speech recognition and conversation phases are performed by AP. Figure 7-Figure 9 To explain, Figure 7 The breath arousal detection phase is exemplified. Figure 8 The speech recognition stage is exemplified. Fig. 9 The dialogue phase is exemplified.

[0184] Figure 7 It is a schematic diagram of the execution process of breath awakening detection in an embodiment of the present application. Figure 7 It mainly explains the relevant steps of the breath awakening detection stage in the entire voice interaction process.

[0185] S701. The breath awakening module starts QMI to register the hand raising event.

[0186] It can be understood that the breath awakening detection module initiates the registration of the hand raising event to the state forwarding module (here taking QMI as an example). Figure 7 Although QMI is used as an example, the breath wake-up module registers and processes the hand-raising event by starting QMI, so that it can monitor and process the "hand-raising event". However, in practice, you can also register the hand-raising event by starting Qsocket, such as using a socket (such as QTcpSocket or QUdpSocket) to monitor and process the "hand-raising event". I will not go into details.

[0187] S702. QMI registers a hand-raising event with the sensor server.

[0188] It should be understood that, as described above, QMI is a message transmission channel, therefore, QMI can work under the scheduling of other modules, for example, the real-time operating system (RTOS) can respond to the breath wake-up module to start the QMI registration hand-raising event request (notification, message), and start and register the hand-raising event with the sensor server through QMI. For example, the real-time operating system can call the registration function to register the hand-raising event, and the registration function can include the event ID, callback function, etc. The return value may be a status code indicating whether the registration is successful.

[0189] S703: The breath awakening module obtains the registration result returned from the sensor server through QMI.

[0190] The returned registration result can be an identification information used to identify whether the hand raising event is successfully registered. The real-time operating system may call the registration function to start QMI to register the hand raising event with the sensor server, and then the sensor server returns the registration result to the real-time operating system through QMI, and then returns it to the breath wake-up module through the real-time operating system. After obtaining the return result, the breath wake-up module determines whether the hand raising event is successfully registered based on the return result.

[0191] S704: When the registration is successful, the breath awakening module turns off the cache function of the audio data access module.

[0192] That is to say, after the breath awakening module determines that the registration of the hand raising event is successful based on the obtained return result in step S703, the cache function of the audio data access module can be immediately closed.

[0193] The closing method can be realized by modifying the flag bit of the audio data access module. The operation of modifying the flag bit can be performed by the breath awakening front module (notification unit) in the breath awakening module.

[0194] It should be understood that closing the cache is to prevent audio data from entering the cache module, so the default state of the audio data access module should be the cache closed state. It will only be turned on when it is necessary to start recording audio data under the trigger of a hand-raising event, so as to store the audio data. When the breath wake-up event is triggered, the AP can read the audio data from the audio data access module for subsequent voice recognition and other operations. For other situations, the audio data caching function does not need to and should not be turned on.

[0195] Steps S701-S704 can be understood as preparation steps, or regarded as "function initialization". After these preparations are completed, breath awakening detection can be performed.

[0196] S705. The breath awakening module controls ADSP to enter low power consumption mode and notifies QMI.

[0197] In one implementation, the breath awakening module or the breath awakening detection module of the breath awakening module calls a function to notify the ADSP to enter the low power consumption mode. For example, the function is called to notify the above-mentioned low power consumption island to enter the low power consumption mode.

[0198] S706. The breath awakening module continues to obtain detection results of the hand raising event in the low power consumption mode.

[0199] Combined with the software architecture example above, the detection of the hand-raising event can be based on the above-mentioned IMU detecting the motion data of the electronic device and then reporting it to the gesture detection module of the breath wake-up module (the gesture detection module is deployed in the low-power memory space of ADSP). The gesture detection module determines whether a hand-raising event occurs (it can be an identifier of a hand-raising event, used to identify the two results of a hand-raising event and a non-hand-raising event). After detecting the hand-raising event, the gesture detection module further reports it to the state forwarding module of the breath wake-up module (the module can be deployed in the non-low-power memory space of ADSP), and the state forwarding module sends the gesture detection result (whether a hand-raising event occurs) to the breath wake-up start module of the breath wake-up module (the module is deployed in the low-power memory space of ADSP). Therefore, the so-called breath wake-up module obtaining the hand-raising event in this step can be executed by the module of the breath wake-up module deployed in the low-power memory space.

[0200] S707. The sensor server reports the hand raising event to the breath wake-up module.

[0201] That is to say, once the gesture detection module detects a hand-raising event or whether the electronic device is in the raised state, the sensor server can report the hand-raising event to the breath wake-up module via QMI. It can be expressed as "if (sensor detects hand-raising action) {exit low power consumption; sensor notifies audio of hand-raising event}", where audio can be regarded as the breath wake-up module.

[0202] S708. When the breath awakening module receives the trigger of the hand raising event, it controls the ADSP to exit the low power consumption mode and notifies the QMI.

[0203] That is to say, after step S708 is executed, the ADSP (or the breath awakening module therein) will operate in the normal working mode (not the low power consumption working mode).

[0204] S709. The breath awakening module opens the cache function of the audio data access module under the triggering of the hand raising event.

[0205] S710: The breath awakening module determines whether a breath awakening event is triggered in the normal working mode.

[0206] In combination with the above, the motion data acquisition unit and the start-up unit of the breath awakening front module of the breath awakening module are operated, and the start-up unit starts the breath awakening detection module, and the breath awakening detection module acquires voice after being started. The motion data acquisition unit transmits the acquired motion data to the breath awakening detection module. The breath awakening detection module can perform breath awakening recognition based on the acquired voice and motion data. The breath awakening detection module can perform breath awakening recognition based on the acquired voice and / or motion data to determine whether a breath awakening event is triggered.

[0207] The judgment result must be one of the two possibilities of "trigger" and "not trigger", that is, one of the two situations of "yes" and "no". Subsequent responses can then be made based on different judgment results. For example, if the judgment result is "trigger", step S716 is executed to wake up the AP for subsequent processing; if the judgment result is "not trigger", steps S717-S719 are executed to end the current round of wake-up process, and continue to switch to low power mode to continuously obtain hand-raising events.

[0208] In the traditional scheme, the complete process of breath wake-up detection ends here. However, since not every hand-raising event is performed by the user for breath wake-up, it may happen that after step S716 is executed, the AP processor finds out after voice recognition that the hand-raising event is not for voice interaction for breath wake-up. For example, the user just picks up the phone and flips it while talking to someone else. In this case, the phone will still be close to the mouth, and voice data that meets the conditions will be collected. Therefore, the AP will not find out that it does not contain questions or instructions until after voice recognition, resulting in a large waste of power consumption; it may also result in that although the judgment result is "not triggered" and steps S707-719 are executed to return to the low power mode, the power consumption during the judgment period is wasted. For example, the user just picked up the phone and quickly put it down, resulting in insufficient voice data to trigger the breath wake-up event, and thus it is judged as "not triggered", but during this period, the ADSP is working in a non-low power mode, which brings unnecessary power consumption waste.

[0209] In response to the above problems, the present application adds a new process, that is, after the breath wake-up module exits the low power mode in step S708, it continues to obtain the hand-raising event, and the sensor server continues to report the non-hand-raising event. Yes, the breath wake-up module can terminate the wake-up process in time (as early as possible) during the execution of step S710 under the triggering of the non-hand-raising event, so as to return the ADSP to the low power working mode as soon as possible and save power consumption.

[0210] In addition, in the present application, when executing step S710, the collected voice data and the detection results of the real-time hand-raising event (the hand-raising event obtained during S710) are combined to determine whether to trigger the breath wake-up event, thereby effectively reducing the power consumption in the non-breath wake-up situation by timely ending the invalid wake-up. Different from this, the traditional solution only combines the hand-raising event reported in S707 (that is, the last hand-raising event reported before exiting the low-power mode) and the collected voice data to determine whether to trigger the breath wake-up event when executing step S710, resulting in a series of power consumption as long as it is a hand-raising event (even if it is not a hand-raising event for breath wake-up), resulting in power consumption waste.

[0211] S711. During the execution of step S710, the sensor server reports the non-hand-raising event to the breath wake-up module.

[0212] It should be understood that the non-hand-raising event here can be understood as the "no" result of the hand-raising event, that is, the detection result of the hand-raising event may be one of the two indications of "yes hand-raising event" and "not hand-raising event", and the so-called "reporting non-hand-raising event" here means that the breath wake-up module obtains a detection result during the period of continuously obtaining the detection results of the hand-raising event, and learns from it that other gestures that are "not hand-raising events" have occurred. The so-called "non-hand-raising event" can be understood as other gestures that are not hand-raising events.

[0213] S712. When the breath awakening module is triggered by a non-hand-raising event, step S710 is ended, thereby ending the current round of breath awakening process.

[0214] That is to say, after receiving the "non-hand-raising event", the breath wake-up module learns that a new posture change has occurred in the current electronic device, causing the "hand-raising event" reported in step S707 to end. Therefore, based on this result, step S710 is ended in advance, thereby ending the current round of breath wake-up process in advance.

[0215] It should be understood that in order to ensure that the wake-up can be achieved, step S710 needs to last for a long time. First, the user needs to wait until the microphone is close enough to the mouth before speaking back, so a certain waiting delay needs to be reserved for the user. Secondly, the user may not speak immediately, and may pause, such as swallowing saliva, clearing the throat, etc., which may also bring possible waiting time, which may bring about the reserved time before recording the voice data. In addition, from the time the user starts speaking to the time the user stops speaking, it is necessary to pause and judge whether the user has finished recording the voice data, and it is also necessary to determine whether to trigger the breath wake-up event based on the collected voice data and the detection results of the real-time hand-raising event, which brings about the duration of step S710 that must be executed. The above-mentioned possible reserved time and the necessary execution time are combined to be the execution time of step S710. Therefore, it can be seen that the execution period of step S710 is a relatively long period of time, generally a few seconds. And the above-mentioned reserved time is often set to a fixed time in traditional solutions, such as 2 seconds or 3 seconds above, etc., which will not be repeated. The present application continues to obtain the hand-raising event during the execution of step S710, so that the posture change of the electronic device can be detected immediately, that is, the interruption of the hand-raising event can be detected in time, so as to end the breath wake-up process and the subsequent AP processing process as soon as possible, saving power consumption.

[0216] S713: When the current round of breath awakening process is finished, the breath awakening module turns off the cache function of the audio data access module.

[0217] S714, the breath awakening module controls ADSP to enter low power consumption mode and notifies QMI.

[0218] This step may refer to step S705 , and the execution content and method of the two may be the same, but the triggering conditions (execution time nodes) are different.

[0219] S715. The breath awakening module continues to obtain detection results of the hand raising event in the low power consumption mode.

[0220] This step may refer to step S706 , and the execution content and method of the two may be the same, but the triggering conditions (execution time nodes) are different.

[0221] S716: When it is determined that a breath awakening event is triggered, the breath awakening module awakens the application processor.

[0222] The subsequent execution process of AP can refer to the above and will not be repeated here.

[0223] S717. When the result of the breath awakening module in executing step S710 is "No", that is, when the breath awakening event is not triggered, the current round of breath awakening process is ended, the ADSP is controlled to enter the low power consumption mode, and the QMI is notified.

[0224] S718. When the current round of breath awakening process is finished, the breath awakening module turns off the cache function of the audio data access module.

[0225] S719. The breath awakening module continues to obtain detection results of the hand raising event in the low power consumption mode.

[0226] This step may refer to steps S706 and S715, and their execution content and method may be the same, but the triggering conditions (execution time nodes) are different.

[0227] Figure 7 The sensor server mainly receives the registration of hand-raising events from audio (which can be regarded as a breath wake-up module), detects hand-raising events, exits low-power mode when a hand-raising action is detected, and notifies audio of the hand-raising event. Audio mainly starts the wake-up word-free function (starts the breath wake-up function), registers hand-raising events with the sensor and calls back the registration results, waits for hand-raising events after successful registration, and when the detection fails (that is, a non-hand-raising event is detected), determines to exit the hand-raising state, notifies the audio data access module (such as the DAM module) to turn off the cache function, interrupts the algorithm processing of ADSP (that is, the step of determining whether to trigger a breath wake-up event), and allows ADSP to enter low-power mode. Audio also needs to obtain hand-raising events in low-power mode, exit low-power mode when it is determined that a hand-raising event has occurred, turn on the cache function and determine whether a breath wake-up event is triggered, stop caching and wake up the relevant APP of the AP for subsequent voice interaction processing when it is determined to be triggered, and turn off the cache and return to low-power mode when it is determined not to be triggered.

[0228] from Figure 7 It can be seen that the present application scheme mainly reduces unnecessary power consumption by continuing to obtain hand-raising events during the execution of determining whether a breath wake-up event is triggered, and terminating the process based on the reported non-hand-raising events.

[0229] Figure 8 It is a schematic flow chart of a speech recognition stage of an embodiment of the present application. Figure 8 The execution processes in the wake-up and non-wake-up scenarios are introduced respectively. Figure 8 The process can be executed under the triggering of step S716.

[0230] S801. The smart assistant of the application layer of the AP fails to call the audio service module (audioserver) of the framework layer, and speech recognition is not started.

[0231] In the embodiment of the present application, after the smart assistant is started, the smart assistant attempts to call the audio service module. Only if the call is successful can the channel between the two be opened and voice recognition be started. When the call fails, the channel cannot be opened and voice recognition cannot be started. Figure 8 The “×” in the figure indicates that the smart assistant failed to call the audio service module (i.e., failed).

[0232] It should be noted that when the smart assistant calls the audio service module, there may be two situations: success and failure. When the audio channel of the audio service module is occupied, or when the smart assistant exits after successful startup, the smart assistant fails to call the audio service module. For example, after the smart assistant is started, the recording function of the electronic device is turned on, and the audio channel of the audio service module is occupied, the smart assistant may fail to call the audio service module. It can be seen that this is mainly because the audio service is not a dedicated module for the voice assistant, nor is it dedicated to voice recognition, so there may be conflicts with other services, so the call may fail.

[0233] S802, the audio trigger module (soundtrigger) of the AP framework layer monitors timeout, and notifies ADSP to return to the breath wake-up detection stage.

[0234] The judgment of the monitoring timeout here can be achieved by setting a time threshold or a timer. In an embodiment of the present application, when the smart assistant successfully calls the audio service module, the smart assistant sends a request to the audio service module to start voice recognition (for example, it can be expressed as the state CaptureState). The audio service module responds to the request to start voice recognition sent by the smart assistant, and sends a request to the audio trigger module of the framework layer to start voice recognition. The audio trigger module of the framework layer starts the timer, so that the audio trigger module can determine whether a request to start breath wake-up recognition is received within the timing time corresponding to the timer. If the request to start voice recognition is not received within the timing time, it means that the smart assistant fails to call the audio service module. If the request to start voice recognition is not received within the timing time, the monitoring times out.

[0235] When notifying ADSP to restart breath wake-up detection, the function StartRecognition can be sent to the audio trigger module of the HAL layer of the AP, and then the audio trigger module of the HAL layer will inform ADSP to switch the state of breath wake-up detection. For example, the state flag state can be changed from BUFFERING to ACTIVE (BUFFERING and ACTIVE correspond to turning off and starting breath wake-up detection respectively).

[0236] Scenario 1 (including steps S801-S802) is mainly aimed at the situation where voice recognition is not performed and the system returns to the breath awakening detection stage due to call failure or monitoring timeout.

[0237] Scenario 2 (including steps S803-S810) is for the case where the call is successful and speech recognition can be performed.

[0238] S803: The smart assistant successfully calls the audio service module and starts voice recognition.

[0239] S804: The audio service module notifies the audio trigger module to start recording.

[0240] In the embodiment of the present application, when the audio service module is called successfully, the audio service module starts voice recognition.

[0241] It should be understood that the recording here is not recording in the traditional sense, but obtaining voice data from the audio trigger module of the HAL layer, and the audio trigger module of the HAL layer obtains the cached voice data from the audio data access module of the ADSP.

[0242] Starting recording can be achieved through the startinput function.

[0243] S805. The audio trigger module of the framework layer informs ADSP to stop breath wake-up detection through the audio trigger module of the HAL layer.

[0244] This can be notified through the StopRecognition function.

[0245] Since the AP now needs to process the speech recognition of the current round of breath-wake-up audio data, it is necessary to let the ADSP pause and continue the breath-wake-up detection. However, it should be understood that step S805 may not inform the ADSP to stop the breath-wake-up detection, because the corresponding strategy can be set on the ADSP side to stop the breath-wake-up detection when the breath-wake-up event is triggered, instead of waiting for the notification from the AP. Therefore, step S805 can be just to inform the audio trigger module of the HAL layer to stop the breath-wake-up detection, so that the data interaction with the audio data access module in the ADSP can be temporarily cut off.

[0246] S806. The audio trigger module of the framework layer reads the recording from the audio trigger module of the HAL layer and forwards it to the smart assistant.

[0247] When reading the recorded LAB data, the audioserver can request the audio trigger module of the HAL layer (primary-hal) to read the data, and then st-hal obtains the recorded data through callback.

[0248] The smart assistant will then perform voice recognition on the recorded audio (voice data) that it has read.

[0249] S807. After the smart assistant completes the voice recognition, the audio service module of the framework layer is notified to stop recording.

[0250] Stopping recording can be achieved through the stopinput function.

[0251] S808. The audio service module further notifies the audio trigger module of the framework layer to stop voice recognition.

[0252] In one example, audioserver may send a request to stop speech recognition to soundtrigger through the function setSundayTriggerCaptureState.

[0253] S809. The audio trigger module of the framework layer informs ADSP to restart breath wake-up detection through the audio trigger module of the HAL layer.

[0254] Soundtrigger may send a request to stop speech recognition to the audio trigger module (st-hal) of the HAL layer through the StopRecognition function. St-hal starts breath wake-up detection and tells ADSP to restart breath wake-up detection.

[0255] exist Figure 8 Before the two scenarios, there can also be a registration phase, in which the main purpose is to prepare the above modules so that the relevant functions of voice recognition in the breath wake-up scenario can be executed. It can be understood that the data path between the smart assistant and each module needs to be opened up so that the voice assistant app can perform voice recognition in the wake-up scenario without the wake-up word. During the registration phase, it is necessary to request ADSP to load the voice recognition model, start gesture detection, trigger the breath wake-up event, read the cache function, report the breath wake-up event, etc.

[0256] It should also be understood that Figure 8 The smart assistant app is just taken as an example, but in practice other applications can also be used to execute it. There is no limitation as long as voice recognition can be performed.

[0257] Fig. 9 It is a schematic flowchart of a dialogue stage of an embodiment of the present application. Fig. 9 It can be understood as being executed during the audio collection phase of the conversation phase of the breath awakening detection phase, that is, it can be understood as a step that can be executed during the execution of step S710.

[0258] S901. The audio service module (Audio service) of the framework layer in the AP determines that the voiceprint recognition is successful and starts the conversation recording.

[0259] After waking up the electronic device without the wake-up word, the voiceprint recognition is automatically performed on the audio data input by the user. After the current user is verified as a legitimate user based on the voiceprint recognition, the device is automatically unlocked and the conversation recording is started. This realizes the voice-controlled unlocking of the electronic device, that is, the contactless unlocking of the electronic device.

[0260] S902: The audio service module of the framework layer requests the audio trigger module in the hardware abstraction layer (HAL layer) of the AP to open an audio input stream.

[0261] The audio service module requests to open the audio input stream based on the success of the voiceprint recognition. The request to establish the stream may include: the audio service module may initialize the audio input stream and return a handle to read the audio data.

[0262] In one example, the audio service module sends a request to open an audio input stream to the primary-hal through the function adev_open_input_stream(); audio_io_handle_thandle.

[0263] Among them, the adev_open_input_stream() function is used to initialize an audio input stream and return a handle of audio_io_handle_thandle type, which can be used to read audio data, control the audio input stream, and configure audio parameters.

[0264] S903: The audio trigger module in the HAL layer creates an audio stream.

[0265] In the embodiment of the present application, the hardware abstraction layer creates an audio stream in response to a request to open an audio stream.

[0266] Exemplarily, the primary-hal may create an audio stream through the function CreateStreamln(), and create a class of StreamlnPrimary() in the method of CreateStreamln().

[0267] S904: The audio service module requests the audio trigger module of the HAL layer to start reading audio data.

[0268] A request to start reading audio data can be sent through the function in_read().

[0269] S905 . The audio trigger module of the HAL layer requests to read audio data from the ADSP.

[0270] It can also be understood that the audio trigger module of the HAL layer controls ADSP to read audio data.

[0271] S906. ADSP obtains real-time audio data from the audio acquisition device (MIC).

[0272] In the embodiment of the present application, the ADSP can read audio data from an audio acquisition device such as a microphone in real time, and perform corresponding algorithm processing on the audio data in the ADSP. For example, the ADSP reads audio data from an audio acquisition device such as a microphone in real time in response to a request for audio data.

[0273] It can be understood that under the trigger of step S707, the recording of breath wake-up detection starts. Once the voice signal input by the user is obtained, steps S901-S905 are immediately executed to complete the preparation for recording, and S906 is executed to record and process the audio data. The so-called ADSP audio data processing is the related processing in step S710, for example, it may include judging whether the breath wake-up requirements are met based on the signal strength of the two microphones, judging whether to trigger a breath wake-up event, and synchronously caching to the audio data access module, etc., which will not be repeated here.

[0274] In order to understand how the present application scheme reduces power consumption, the following combination Fig.10 Explanation is given with specific examples.

[0275] Fig.10 This is a comparison chart of power consumption of the embodiment of the present application and the traditional solution in the same interaction scenario.

[0276] like Fig.10 (a) and (b) show power consumption diagrams of the conventional solution and the solution of the present application, respectively, where the horizontal axis represents the natural time t and the vertical axis represents the power consumption P.

[0277] Assumptions Fig.10 The scenario is: the user picks up the stationary electronic device A from the desktop at time T1, and puts it down immediately after about 0.3 seconds (s). The user does not speak, and then turns over the electronic device A at time T2, which takes about 0.3s. The user does not speak. Then at time T3, while chatting with someone, picks up the electronic device B, brings it close to the face, turns it over, and uses the small mirror on the back protective cover to take a picture of the face. Assuming that the flipping occurs at 0.5s, the screen of electronic device A is off at times T1, T2 and T3, and the breath wake-up function is activated in electronic device A. The breath wake-up detection uses ADSP, and the gesture detection and breath wake-up startup detection are deployed in the low-power space of ADSP. The breath wake-up detection algorithm and audio data access module are deployed in the non-low-power space of ADSP. The AP is used to perform voice recognition and subsequent responses, and the fixed waiting time for collecting voice data is 2 seconds. Fig.10(a) and (b) are the power consumption in the same scenario as above.

[0278] In the traditional solution, once a hand-raising event is detected, the ADSP will be awakened to determine whether a breath wake-up event is triggered. During this period, user voice data will be collected. If the user sound signal is collected within 2 seconds, the user voice data will be continuously collected and a judgment will be made as to whether a breath wake-up event is triggered. When it is determined to be triggered, the AP will be awakened for subsequent processing such as voice recognition. If no qualified voice signal is collected within 2 seconds, the system will return to low power consumption mode.

[0279] like Fig.10 As shown in (a), ADSP wakes up at time T1, and the power rises from P1 to P2, and starts trying to collect user voice data. Since the user does not speak, ADSP determines that the breath wake-up event is not triggered after 2s, and thus returns to power P1; ADSP wakes up at time T2, and the power rises from P1 to P2, and starts trying to collect user voice data. Since the user does not speak, ADSP determines that the breath wake-up event is not triggered after 2s, and thus returns to power P1; ADSP wakes up at time T3, and the power rises from P1 to P2, and starts trying to collect user voice data. Since the user is chatting, voice data can be collected when it gets close to the user's mouth. Assuming that voice data is collected continuously for 1.8s from 0.5s, ADSP power further rises from P2 to P3 and continues for 1.8s. After that, ADSP Based on the 1.8s voice data, DSP determines that a breath wake-up event can be triggered, and then further wakes up AP for voice recognition. The power further rises to P4 (ADSP power + AP power, but AP power may be much higher than ADSP power). Since AP voice recognition needs to call the audio service module and wait, it is assumed that AP may need to wait for a while for recognition. Suppose that the recognized text information after 2.3s makes AP confirm that it cannot respond, thus ending the current round of voice interaction, AP returns to low power mode, and notifies ADSP to also return to low power mode and return to low power mode for breath wake-up detection. Therefore, Fig.10 The total power consumption of (a) is: P1*total running time+P2*2s+P2*2s+P2*0.5s+P3*1.8s+P4*2.3s.

[0280] It should also be understood that the difference between P3 and P2 may be almost negligible, because during these 1.8s and the previous 0.5s, the ADSP is operating in normal working mode, and the AP has not been awakened. Therefore, whether there is a continued increase in power consumption depends on the difficulty of processing the collected voice data.

[0281] In the present application scheme, once a hand-raising event is detected, the ADSP will be awakened to determine whether a breath wake-up event is triggered. During this period, user voice data will be collected. If the user sound signal is collected within 2 seconds, the user voice data will be continuously collected and a judgment will be made as to whether a breath wake-up event is triggered. When it is determined to be triggered, the AP will be awakened for subsequent processing such as voice recognition. If no qualified voice signal is collected within 2 seconds, the mode will return to low power consumption mode. However, on this basis, the detection results of the hand-raising event will continue to be obtained. Once an indication of the end of the hand-raising event is received, the judgment process and all subsequent processes will be terminated directly.

[0282] like Fig.10 As shown in (b), ADSP wakes up at time T1, and the power rises from P1 to P2, and starts to try to collect user voice data. Since the hand-raising event ends after 0.3s, ADSP ends the process based on the detection result of the new hand-raising event, and returns to power P1; ADSP wakes up at time T2, and the power rises from P1 to P2, and starts to try to collect user voice data. Since the hand-raising event ends after 0.3s, ADSP ends the process based on the detection result of the new hand-raising event, and returns to power P1; At time T3, ADSP wakes up, the power rises from P1 to P2, and starts to try to collect user voice data. Since the hand-raising event ends after 0.5s, ADSP ends the process based on the detection result of the new hand-raising event, and returns to power P1. Therefore, Fig.10 The total power consumption in (b) is: P1*total running time+P2*0.4s+P2*0.3s+P2*0.5s.

[0283] From the comparison diagram, it can be seen that the present application scheme "purifies" the hand-raising event, eliminating the influence of hand-raising events that are not used for breath awakening, and can effectively reduce power consumption. However, it should be understood that Fig.10 It is only for the purpose of illustrating how the solution of the present application reduces power consumption in the same scenario. There is no limitation on the examples and specific values ​​of the specific scenarios.

[0284] The above mainly introduces the method of the embodiment of the present application in conjunction with the accompanying drawings. It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence, these steps are not necessarily performed in sequence in the order shown in the figure. Unless clearly stated in this article, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a part of the steps or stages in other steps. The device of the embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0285] Fig.11 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Fig.11 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Fig.11 As shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, an audio module 270, a speaker 270A, a receiver 270B, a microphone N70C, an earphone interface 270D, a sensor module 280, a camera 293, a display screen 294, etc. Among them, the sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an acceleration sensor 280E, etc.

[0286] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0287] For example, Fig.11The processor 210 shown may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), an audio digital signal processor (ADSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0288] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0289] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0290] In the embodiment of the present application, the AP and ADSP mainly work together to complete the voice interaction in the breath wake-up scenario. In addition, in view of the limited low-power memory space of ADSP, the present application subdivides the breath wake-up module into multiple submodules (such as gesture detection module, breath wake-up startup module, etc.), and deploys them in the low-power memory space and non-low-power memory space of ADSP respectively. In particular, the relevant algorithms for breath wake-up detection are deployed in the non-low-power memory space, so that even low-end and mid-end processors can deploy breath wake-up detection solutions. On this basis, when judging whether to trigger a breath wake-up event, an additional layer of screening conditions is added to eliminate unnecessary power consumption caused by non-breath wake-up hand-raising events, so as to further reduce power consumption.

[0291] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0292] In some embodiments, the I2S interface can be used for audio communication. The processor 210 can include multiple I2S buses. The processor 210 can be coupled to the audio module 270 via the I2S bus to achieve communication between the processor 210 and the audio module 270.

[0293] In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 through the I2S interface to implement the function of answering calls through a Bluetooth headset.

[0294] In some embodiments, the PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. The audio module 270 and the wireless communication module 260 can be coupled via a PCM bus interface.

[0295] In some embodiments, the audio module 270 can also transmit audio signals to the wireless communication module 260 through the PCM interface to implement the function of answering calls through a Bluetooth headset. It should be understood that both the I2S interface and the PCM interface can be used for audio communication.

[0296] In some embodiments, the UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. The UART interface is generally used to connect the processor 210 and the wireless communication module 260. For example, the processor 210 communicates with the Bluetooth module in the wireless communication module 260 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 270 can transmit an audio signal to the wireless communication module 260 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0297] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0298] The electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0299] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0300] In the embodiment of the present application, the display screen 294 can be used to display various interfaces, such as interface 101-interface 108.

[0301] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0302] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0303] In the embodiment of the present application, it is mainly used for performing speech recognition and feeding back corresponding results to the user based on the speech recognition results.

[0304] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0305] In the embodiment of the present application, the relevant models of the breath awakening detection stage need to be deployed in the non-low-power memory space of the ADSP, while other models such as the speech recognition model and the question-answering model can be deployed in the memory space of the AP or the internal memory 221. The applet code of the relevant execution process can also be stored in the internal memory 221.

[0306] The electronic device 200 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone N70C and the application processor.

[0307] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

[0308] The speaker 270A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 200 can listen to music or listen to a hands-free call through the speaker 270A.

[0309] The receiver 270B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 200 receives a call or voice message, the voice can be received by placing the receiver 270B close to the human ear.

[0310] The microphone N70C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone N70C to input the sound signal into the microphone N70C. The electronic device 200 can be provided with at least one microphone N70C. In other embodiments, the electronic device 200 can be provided with two microphones N70C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 200 can also be provided with three, four or more microphones N70C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0311] In the embodiment of the present application, the sound signal of the voice is collected by a microphone, and then the ADSP determines whether to trigger a breath wake-up event based on the collected sound signal, and after the triggering, the AP performs voice recognition and further responds to the collected sound signal.

[0312] The pressure sensor 280A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 280A can be set on the display screen 294. There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 280A, the capacitance between the electrodes changes. The electronic device 200 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 294, the electronic device 200 detects the touch operation intensity according to the pressure sensor 280A. The electronic device 200 can also calculate the touch position according to the detection signal of the pressure sensor 280A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0313] In the embodiment of the present application, touch screen operations such as clicks in interfaces 101-108 can be collected by pressure sensor 280A.

[0314] The gyro sensor 280B can be used to determine the motion posture of the electronic device 200. In some embodiments, the angular velocity of the electronic device 200 around three axes (i.e., the x, y, and z axes) can be determined by the gyro sensor 280B. The gyro sensor 280B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 280B detects the angle of the electronic device 200 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 200 through reverse movement to achieve anti-shake. The gyro sensor 280B can also be used for navigation and somatosensory game scenes.

[0315] The acceleration sensor 280E can detect the magnitude of the acceleration of the electronic device 200 in all directions (generally three axes). When the electronic device 200 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0316] In the embodiment of the present application, the motion data of the electronic device can be detected with the help of the gyroscope sensor 280B, the acceleration sensor 280E and / or other motion sensors, so that it can be determined whether a hand-raising event has occurred based on the motion data.

[0317] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0318] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0319] An embodiment of the present application also provides an electronic device, which includes: multiple processors, a memory, and a computer program stored in the memory and executable on multiple processors, wherein when the multiple processors execute the computer program, the electronic device can implement the steps in any of the above methods.

[0320] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by an electronic device, the steps in the above-mentioned method embodiments can be implemented.

[0321] Computer-readable media may include at least: any entity or device that can carry computer program codes to a camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disk. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0322] The present application embodiment provides a computer program product, which includes a computer program, and when the computer program is executed by an electronic device, the steps in the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form.

[0323] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0324] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0325] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0326] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0327] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0328] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0329] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0330] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0331] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A voice interaction method, applied to an electronic device, characterized in that: include: At a first moment, the electronic device is lifted, and the screen of the electronic device is off, and the power consumption of the electronic device changes from a first value to a second value, and before the first moment, the screen of the electronic device is off, the electronic device is not lifted, and the second value is greater than the first value; At a second moment after the first moment, the electronic device is not lifted, and the screen of the electronic device is in an off state, and the power consumption of the electronic device changes from the second value to the first value.

2. The method according to claim 1, characterized in that At a second moment after the first moment, the electronic device is not lifted, and the screen of the electronic device is in an off state, and the power consumption of the electronic device changes from the second value to the first value, including: At the second moment, based on the electronic device being converted from the lifted state to the unlifted state, the power consumption of the electronic device is changed from the second value to the first value.

3. The method according to claim 1, characterized in that The method further comprises: Before the first moment, the first processor of the electronic device continuously detects a hand-raising event in a low power consumption mode, wherein the hand-raising event is used to indicate whether the electronic device is lifted, wherein when a hand-raising event is detected, it indicates that the electronic device is lifted, and when a non-hand-raising event is detected, it indicates that the electronic device is not lifted; At the first moment, in response to detecting a hand raising event, the first processor switches from a low power consumption mode to a normal working mode, and starts to determine whether a breath awakening event is triggered in the normal working mode, and continues to obtain a detection result of the hand raising event in the normal working mode; At the second moment, in response to detecting a non-hand-raising event, the first processor switches from a normal operating mode to a low power consumption mode, and continues to detect hand-raising events in the low power consumption mode.

4. The method according to claim 1, characterized in that The method further comprises: Between the first moment and the third moment, the electronic device remains lifted, and the screen of the electronic device is in an off state, the time interval between the first moment and the second moment is smaller than the time interval between the first moment and the third moment, and the time interval between the first moment and the third moment is a preset time interval; At the third moment, the power consumption of the electronic device changes from the second value to the first value.

5. The method according to claim 4, characterized in that At the third moment, the power consumption of the electronic device changes from the second value to the first value, including: At the third moment, based on the fact that the electronic device has not collected voice data that meets the first preset condition, the power consumption of the electronic device changes from the second value to the first value, and the first preset condition includes: the signal strength of the voice data collected by the first microphone of the electronic device is greater than or equal to the first preset signal strength threshold, the signal strength of the voice data collected by the second microphone of the electronic device is greater than or equal to the second preset signal strength threshold, and the difference between the signal strength of the voice data collected by the first microphone and the signal strength of the voice data collected by the second microphone is greater than or equal to the preset signal strength difference threshold.

6. The method according to claim 5, characterized in that At the third moment, based on the fact that the electronic device does not collect voice data that meets the first preset condition, the power consumption of the electronic device changes from the second value to the first value, including: Before the first moment, the first processor of the electronic device operates in a low power consumption mode; between the first moment and the third moment, the first processor operates in a normal working mode; The first processor continuously reads the voice data collected by the first microphone and the second microphone from the first moment; At the third moment, the first processor determines that the breath wake-up event is not triggered based on the fact that the electronic device is lifted up and the electronic device has not collected voice data that meets the first preset condition, controls the first processor to switch from the normal working mode to the low power consumption mode, and continues to detect the hand raising event in the low power consumption mode.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: At a fifth moment after the first moment, the screen of the electronic device is turned on, and the power consumption of the electronic device is a third value, which is greater than the second value.

8. The method according to claim 7, characterized in that The step of turning on the screen of the electronic device at a fifth moment after the first moment, and the power consumption of the electronic device being a third value, wherein the third value is greater than the second value, comprises: At the first moment, the first processor of the electronic device switches from a low power consumption mode to a normal operating mode, so that the power consumption of the electronic device increases from the first value to the second value; At a fourth moment after the first moment, based on the electronic device starting to collect voice data that meets a first preset condition, the first processor starts to continuously read the collected voice data; the first preset condition includes: the signal strength of the voice data collected by the first microphone of the electronic device is greater than or equal to a first preset signal strength threshold, the signal strength of the voice data collected by the second microphone of the electronic device is greater than or equal to a second preset signal strength threshold, and the difference between the signal strength of the voice data collected by the first microphone and the signal strength of the voice data collected by the second microphone is greater than or equal to a preset signal strength difference threshold; At the fifth moment after the fourth moment, the first processor completes reading the first voice data, and the first processor wakes up the second processor based on that the first voice data satisfies the second preset condition, so that the second processor switches from the low power consumption mode to the normal working mode and performs voice recognition and response to the first voice data in the normal working mode, thereby increasing the power consumption of the electronic device from the second value to the third value. Satisfying the second preset condition is used to indicate that the first voice data can be used to trigger a breath wake-up event.

9. The method according to claim 8, characterized in that The method further comprises: At the fifth moment, based on the fact that the first voice data does not satisfy the second preset condition, the first processor switches from the normal working mode to the low power consumption mode, so that the power consumption of the electronic device changes from the second value to the first value.

10. The method according to claim 8, characterized in that The method further comprises: At a sixth moment after the fifth moment, the screen of the electronic device changes from the on state to the off state, and the power consumption of the electronic device changes from the third value to the first value.

11. The method according to claim 10, characterized in that At a sixth moment after the fifth moment, the screen of the electronic device changes from a light-on state to an off state, and the power consumption of the electronic device changes from the third value to the first value, including: At the sixth moment, based on the second processor completing the response to the first voice data, the first processor and the second processor both switch to a low power consumption mode, so that the power consumption of the electronic device changes from the third value to the first value.

12. The method according to claim 8, characterized in that The method further comprises: At the fourth moment, based on the fact that the electronic device has not collected voice data that meets the first preset condition, the first processor has not read voice data that meets the first preset condition, and the electronic device continues to collect voice data until the third moment, the time interval between the third moment and the first moment is the preset time interval.

13. A chip system, applied to electronic equipment, characterized in that: The chip system includes a first processor and a second processor, the first processor is used to perform breath awakening detection, and the second processor is used to respond to voice interaction with voice data collected by the second processor when awakened by the first processor; At a first moment, the electronic device is lifted up and the screen of the electronic device is in an off state, the first processor triggers the power consumption of the electronic device to change from a first value to a second value, before the first moment, the screen of the electronic device is in an off state, the electronic device is not lifted up, and the second value is greater than the first value; At a second moment after the first moment, the electronic device is not lifted, and the screen of the electronic device is in an off state, and the power consumption of the electronic device changes from the second value to the first value.

14. An electronic device, characterized in that: The electronic device includes a memory, multiple processors, a sensor, an audio acquisition device and a display screen, the sensor is used to collect motion data of the electronic device, the audio acquisition device is used to collect voice of the electronic device, the display screen is used to display an interface, the memory is used to store instructions, and the multiple processors are used to run the instructions stored in the memory, so that the electronic device executes the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 12.

Images