Voice interaction method, chip system, electronic device, and storage medium

Abstract

The embodiment of the application provides a voice interaction method, a chip system, an electronic device and a storage medium, and relates to the technical field of voice interaction. The voice interaction method is applied to an electronic device, the electronic device comprises a first microphone and a second microphone, and the method comprises the following steps: at a first time, the electronic device is lifted, the screen of the electronic device is in an off state, the power consumption of the electronic device changes from a first value to a second value, before the first time, the screen of the electronic device is in the off state, the electronic device is not lifted, the second value is greater than the first value, between the first time and a second time, the electronic device receives a first voice, at the second time, the electronic device is turned on, and after the second time, the power consumption of the electronic device is a third value, and the third value is greater than the second value. The power consumption of the electronic device can be ensured to be low when breath awakening, and the universality of the breath awakening mode is improved.

Description

Technical Field

[0001] This application relates to the field of voice interaction technology, specifically to a voice interaction method, a chip system, an electronic device, and a storage medium. Background Technology

[0002] If a user wants to interact with an electronic device via voice, they can wake up the voice interaction module using breath. Breath wake-up does not require the user to output a specific wake-up word. However, current breath wake-up methods consume power. Therefore, how to wake up the voice interaction module via breath while maintaining low power consumption is a pressing issue, especially when the screen is off and the application processor (AP) is in sleep mode. Summary of the Invention

[0003] This application provides a voice interaction method, a chip system, an electronic device, and a storage medium, which can ensure low power consumption of the electronic device and improve the universality of the breath wake-up method.

[0004] In a first aspect, a voice interaction method is provided, applied to an electronic device, the electronic device including a first microphone and a second microphone, the method comprising: at a first moment, the electronic device is lifted and the screen of the electronic device is off, the power consumption of the electronic device changes from a first value to a second value; before the first moment, the screen of the electronic device is off, the electronic device is not lifted, and the second value is greater than the first value; between the first moment and the second moment, the electronic device receives a first voice; at a second moment, the screen of the electronic device is turned on; after the second moment, the power consumption of the electronic device is a third value, the third value being greater than the second value.

[0005] In this embodiment, before the first moment, the screen of the electronic device is off and the main processor is in sleep mode, at which time the power consumption of the electronic device is a first value. After the electronic device is lifted, the first voice is received. Optionally, the electronic device being lifted triggers the reception of the first voice. Optionally, the above-mentioned reception of the first voice can be understood as: after the electronic device is lifted, the first voice is stored in the non-low-power storage space of the second processor of the electronic device. Based on receiving the first voice after the electronic device is lifted, the power consumption of the electronic device is reduced. From before the first moment until the detection of the electronic device being lifted, and then from the detection of the electronic device being lifted until the third moment when the user wakes up the main processor through the first voice and the screen of the electronic device lights up, the power consumption of the electronic device gradually increases, that is, from the first value to the second value, and then from the second value to the third value, thereby realizing that when the electronic device is woken up by breath, the power consumption of the electronic device gradually increases, ensuring that the power consumption of the electronic device is low.

[0006] Optionally, the method further includes: after the second moment, the electronic device screen is turned off; at the third moment, the electronic device is lifted, and the screen is off, the power consumption of the electronic device changes from a first value to a third value; before the third moment, the screen is off, the electronic device is not lifted, and the third value is greater than the first value; between the third and fourth moments, the electronic device does not receive voice; at the fourth moment, the power consumption of the electronic device is the first value. Thus, when the electronic device does not receive voice, its power consumption decreases, further ensuring low power consumption.

[0007] Optionally, the method further includes: after the second moment, the electronic device screen is turned off; at the fifth moment, the electronic device is lifted, and the screen of the electronic device is off, the power consumption of the electronic device changes from a first value to a fourth value; before the fifth moment, the screen of the electronic device is off, the electronic device is not lifted, and the fourth value is greater than the first value; between the fifth and sixth moments, the electronic device receives second voice, and the first distance between the sound source of the second voice and the second microphone is greater than or equal to the second distance between the sound source of the second voice and the first microphone; at the sixth moment, the power consumption of the electronic device is the first value. Thus, when the voice received by the electronic device does not meet the preset conditions, the power consumption of the electronic device decreases, further ensuring that the power consumption of the electronic device is low.

[0008] Optionally, the method further includes: between a first moment and a second moment, the electronic device receives a first voice message, wherein a third distance between the sound source of the first voice message and the second microphone is less than a fourth distance between the sound source of the first voice message and the first microphone; at the second moment, the electronic device lights up its screen, including: based on the third distance being less than the fourth distance, the electronic device lights up its screen at the second moment; and displays a first interface, the first interface being related to the first voice message and including the content of the first voice message. Thus, when the voice message received by the electronic device meets preset conditions, the electronic device lights up its screen and displays the first interface, successfully waking up the voice interaction module.

[0009] Optionally, at a second moment, the electronic device lights up its screen, including: based on the first voice message including human voice, the electronic device lights up its screen at the second moment; and displays a first interface, the first interface being related to the first voice message and including the content of the first voice message. Thus, when the voice message received by the electronic device meets preset conditions, the electronic device lights up its screen and displays the first interface, successfully waking up the voice interaction module.

[0010] Optionally, prior to the first moment, the method further includes: displaying a second interface, the second interface including a first control, the first control being displayed in a first state; in response to an operation on the first control, the first control being displayed in a second state, the first control being displayed in the second state indicating that the wake-up word-free wake-up function of the electronic device is enabled. Thus, the electronic device can be instructed to wake up the voice interaction module without a wake-up word by controlling the first control.

[0011] Optionally, before the first moment, the screen of the electronic device is off and the electronic device is not lifted, including: before the first moment, the screen of the electronic device is off and the electronic device is not lifted, the first processor of the electronic device is in a sleep state, and the power consumption of the electronic device is a first value; when the electronic device is lifted and the screen of the electronic device is off, the power consumption of the electronic device changes from the first value to a second value, including: when the electronic device is lifted and the screen of the electronic device is off, the first processor remains in a sleep state, the operating voltage and / or operating current of the first memory of the second processor of the electronic device increases, and based on the increase in the operating voltage and / or operating current of the first memory, the power consumption of the electronic device changes from the first value to the second value.

[0012] Optionally, before the first moment, the screen of the electronic device is off and the electronic device is not lifted, including: before the first moment, the screen of the electronic device is off and the electronic device is not lifted, the first processor of the electronic device is in a sleep state, and the second processor of the electronic device is in a low-power mode, and the power consumption of the electronic device is a first value based on the first processor being in a sleep state and the second processor being in a low-power mode; at the first moment, the electronic device is lifted and the screen of the electronic device is off, and the power consumption of the electronic device changes from the first value to a second value, including: at the first moment, the screen of the electronic device is off and the first processor is in a sleep state, and in response to the electronic device being lifted, the second processor exits the low-power mode, and the power consumption of the electronic device changes from the first value to the second value based on the second processor exiting the low-power mode.

[0013] Optionally, the second processor being in a low-power mode includes: the first memory of the second processor being in a low-power state, the operating voltage of the first memory being a first voltage, and the operating current of the first memory being a first current; the second processor exiting the low-power mode includes: the first memory being in a non-low-power state, the operating voltage of the first memory being a second voltage, and the operating current of the first memory becoming a second current, wherein the second voltage is greater than the first voltage, and / or the second current is greater than the first current.

[0014] Optionally, at the second moment, the electronic device turning on its screen includes: at the second moment, the first processor of the electronic device is in a working state, the electronic device turns on its screen, and based on the first processor being in a working state, the power consumption of the electronic device is a third value.

[0015] Optionally, before the first moment, the method further includes: controlling the second processor of the electronic device to enter a low-power mode; and assigning a first value to the flag of the audio data cache to disable the audio data cache.

[0016] Optionally, the process of raising the electronic device at the first moment further includes: detecting a hand-raising event at the first moment, determining that the electronic device has been raised in response to the hand-raising event, controlling the second processor to exit the low-power mode; and assigning a second value to the flag bit of the audio data cache to open the audio data cache.

[0017] Optionally, between the first and second moments, the electronic device receiving the first voice includes: between the first and second moments, the electronic device receiving the first voice and storing the first voice in an audio data cache; the first microphone collecting the signal strength of the first voice as a first signal strength; the second microphone collecting the signal strength of the first voice as a second signal strength; the difference between the second signal strength and the first signal strength as a first difference; determining whether the first difference is greater than a first threshold, whether the second signal strength is greater than the second threshold, and whether the first voice includes human voice; at the second moment, the electronic device turning on the screen includes: at the second moment, based on the first difference being greater than the first threshold, the second signal strength being greater than the second threshold, and the first voice including human voice, controlling the first processor of the electronic device to exit the sleep state, and assigning the flag bit of the audio data cache to the first value to close the audio data cache, and the electronic device turning on the screen.

[0018] Optionally, between the third and fourth time points, the electronic device not receiving voice includes: between the third and fourth time points, in response to not receiving voice, the electronic device controls the second processor to enter a low-power mode and assigns the first value to the flag bit of the audio data buffer to disable the audio data buffer.

[0019] Optionally, between the fifth and sixth moments, the electronic device receives the second voice, and the third distance between the sound source of the second voice and the second microphone is greater than or equal to the fourth distance between the sound source of the second voice and the first microphone, including: between the fifth and sixth moments, the electronic device receives the second voice; the first microphone collects the signal strength of the second voice as a third signal strength, the second microphone collects the signal strength of the second voice as a fourth signal strength, and the difference between the fourth signal strength and the third signal strength is a second difference; based on the second difference being less than or equal to a first threshold, the second processor is controlled to enter a low-power mode, and the flag bit of the audio data cache is assigned a first value to disable the audio data cache.

[0020] Optionally, the electronic device includes a voice interaction module and a preset wake-up module. A first processor of the electronic device is used to run the voice interaction module, and a second processor of the electronic device is used to run the preset wake-up module. The preset wake-up module includes a gesture detection module, a breath wake-up activation module, and a breath wake-up detection module. The gesture detection module is used to detect the motion state of the electronic device. The breath wake-up activation module is used to activate the breath wake-up detection module when the electronic device is lifted. The breath wake-up detection module is used to determine whether to wake up the first processor based on the received voice. The breath wake-up detection module is deployed in the first memory of the second processor. The gesture detection module and the breath wake-up activation module are deployed in the second memory of the second processor.

[0021] Optionally, before the first moment, the screen of the electronic device is off and the electronic device is not lifted. The second value being greater than the first value includes: before the first moment, the gesture detection module and the breath wake-up activation module are running on the second processor, and the breath wake-up detection module is not running on the second processor because the electronic device is not lifted; based on the gesture detection module and the breath wake-up activation module running on the second processor, but the breath wake-up detection module not running on the second processor, the first processor is in a sleep state, and the power consumption of the electronic device is the first value; at the first moment, the electronic device is lifted and the screen of the electronic device is off, and the power consumption of the electronic device changes from the first value to the second value includes: at the first moment, based on the electronic device being lifted, the gesture detection module detects a hand-raising event; the gesture detection module transmits the hand-raising event to the breath wake-up activation module; the breath wake-up activation module activates the breath wake-up detection module based on the hand-raising event, and the breath wake-up detection module runs on the second processor; based on the second processor running the breath wake-up detection module, the first processor is in a sleep state, and the power consumption of the electronic device changes from the first value to the second value.

[0022] Optionally, the preset wake-up module further includes a state forwarding module, which is deployed in the first memory of the second processor. At a first moment, in response to the electronic device being lifted, the gesture detection module detects a hand-lifting event and transmits the hand-lifting event to the breath wake-up activation module, including: at the first moment, based on the electronic device being lifted, the gesture detection module detects a hand-lifting event; the gesture detection module transmits the hand-lifting event to the state forwarding module; the state forwarding module transmits a lift-up state to the breath wake-up activation module based on the hand-lifting event; and the breath wake-up activation module activates the breath wake-up detection module based on obtaining the lift-up state.

[0023] Optionally, the preset wake-up module further includes a breath wake-up pre-module and an audio data storage module, with the audio data storage module deployed in the first memory; the breath wake-up pre-module runs on the second processor; the breath wake-up activation module, based on obtaining a lift-off state, activates the breath wake-up detection module by: the breath wake-up activation module triggering the breath wake-up pre-module to run based on obtaining a lift-off state; the breath wake-up pre-module controlling the second processor to exit low-power mode; the breath wake-up pre-module activating the breath wake-up detection module; the breath wake-up pre-module acquiring motion data collected by the sensors of the electronic device and transmitting the motion data to the breath wake-up detection module; and the breath wake-up pre-module assigning a second value to the flag bit of the audio data storage module to control the audio data storage module to perform audio data caching.

[0024] Optionally, between the first moment and the second moment, the electronic device receiving the first voice includes: between the first moment and the second moment, the audio data access module, in response to a flag bit being a second value, storing the first voice collected by the sensors of the electronic device.

[0025] Optionally, the breath wake-up pre-module includes an exit low-power unit, a motion data acquisition unit, a notification unit, and a startup unit; the exit low-power unit is deployed in the second memory, and the motion data acquisition unit, notification unit, and startup unit are deployed in the first memory; the exit low-power unit controls the second processor to exit the low-power mode; the motion data acquisition unit acquires motion data and transmits the motion data to the breath wake-up detection module; the notification unit assigns a second value to the flag bit of the audio data access module; and the startup unit starts the breath wake-up detection module.

[0026] Secondly, a chip system is provided for use in an electronic device. The chip system includes a first processor and a second processor. At a first moment, the electronic device is lifted and its screen is off. The second processor triggers the power consumption of the electronic device to change from a first value to a second value. Before the first moment, the screen of the electronic device is off and the electronic device is not lifted, so the second value is greater than the first value. Between the first and second moments, the second processor receives first voice data. At the second moment, the second processor wakes up the first processor. After the second moment, the first processor controls the screen of the electronic device to turn on, and the power consumption of the electronic device is a third value, which is greater than the second value.

[0027] The first voice data can be understood as the voice data of the first voice mentioned above.

[0028] Optionally, after the second moment, the first processor controls the electronic device to turn off its screen; at the third moment, the electronic device is lifted and its screen is off, and the second processor triggers the power consumption of the electronic device to change from the first value to the third value. Before the third moment, the screen of the electronic device is off, the electronic device is not lifted, and the third value is greater than the first value; between the third moment and the fourth moment, the second processor does not receive voice data; at the fourth moment, the second processor triggers the power consumption of the electronic device to the first value.

[0029] Optionally, after the second moment, the electronic device screen is turned off; at the fifth moment, the electronic device is lifted, and the screen of the electronic device is off, and the second processor triggers the power consumption of the electronic device to change from the first value to the fourth value. Before the fifth moment, the screen of the electronic device is off, the electronic device is not lifted, and the fourth value is greater than the first value; between the fifth moment and the sixth moment, the second processor receives the second voice, and the first distance between the sound source of the second voice and the second microphone is greater than or equal to the second distance between the sound source of the second voice and the first microphone; at the sixth moment, the second processor triggers the power consumption of the electronic device to change to the first value.

[0030] Optionally, between the first and second moments, the second processor receives the first voice data, and the third distance between the sound source of the first voice data and the second microphone is less than the fourth distance between the sound source of the first voice data and the first microphone; at the second moment, the electronic device lights up its screen, including: based on the third distance being less than the fourth distance, the first processor controls the electronic device to light up its screen at the second moment; and displays a first interface, the first interface including the content of the first voice.

[0031] Optionally, at a second moment, the electronic device lights up its screen, including: based on first voice data including human voice, a first processor controls the electronic device to light up its screen at the second moment; and displays a first interface, the first interface including the content of the first voice.

[0032] Thirdly, an electronic device is provided, comprising a memory, a processor, a sensor, an audio acquisition device, and a display screen. The sensor is used to acquire motion data of the electronic device, the audio acquisition device is used to acquire voice data of the electronic device, the display screen is used to display an interface, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory, causing the electronic device to perform any of the above-mentioned voice interaction methods.

[0033] Fourthly, a computer-readable storage medium is provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the voice interaction methods described above.

[0034] The technical effects brought about by the second, third and fourth aspects of the embodiments of this application can be found in the relevant description of the voice interaction method of the first aspect above, and will not be repeated here. Attached Figure Description

[0035] Figures 1A to 1H This is a schematic diagram illustrating an application scenario of the voice interaction method provided in this application embodiment.

[0036] Figure 2 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0037] Figure 3 This is a schematic diagram of the sound processing device provided in the embodiments of this application.

[0038] Figure 4 This is a schematic diagram of the software structure of the electronic device provided in the embodiments of this application.

[0039] Figure 5 This is a schematic diagram of the workflow of the preset wake-up module provided in the embodiments of this application.

[0040] Figures 6A to 6C This is a power consumption diagram of the electronic device provided in the embodiments of this application.

[0041] Figure 7 This is a schematic diagram of the startup phase of the voice interaction method provided in the embodiments of this application.

[0042] Figure 8 This is a schematic diagram of the breath wake-up recognition method provided in the embodiments of this application.

[0043] Figure 9 This is a schematic diagram of the recognition stage of the voice interaction method provided in the embodiments of this application.

[0044] Figure 10 This is a schematic diagram of the dialogue phase of the voice interaction method provided in the embodiments of this application.

[0045] Figure 11 This is a schematic diagram of a voice interaction method provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0047] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0048] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0049] The term "user interface (UI)" used in the following embodiments of this application refers to a medium that facilitates interaction between a user and an application or operating system, enabling the conversion between the internal form of information and a form acceptable to the user. A user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on an electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can include visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, and navigation bars displayed on the screen of an electronic device.

[0050] Voice interaction is a human-computer interaction method where electronic devices can perform corresponding actions based on user-inputted voice commands, such as querying information, playing music, or activating a function. To interact with an electronic device via voice, the user must first activate its voice interaction module. Methods for activating the voice interaction module include: shortcut key activation, wake-up word activation, and breath activation. Wake-up word activation requires the user to input a specific wake-up word. Breath activation allows the user to activate the device or application by uttering certain sounds (such as voice commands, breathing sounds, coughing sounds, or other non-specific words) without needing to input a specific wake-up word. Breath activation can also be called wake-up word-free activation; this application does not specifically limit its name.

[0051] The inventors, through implementing the embodiments of this application, discovered that in the prior art, breath wake-up methods require continuous monitoring or periodic checks of signals (such as voice signals) to determine whether to wake up the voice interaction module. When breath wake-up requires the participation of the main processor, such as the application processor (AP), even low-frequency periodic checks may frequently wake up the main processor, which conflicts with the main processor's sleep state, leading to increased power consumption of the electronic device. To achieve breath wake-up of the voice interaction module when the electronic device screen is off and the main processor, such as the AP, is in sleep mode, while ensuring low power consumption, the embodiments of this application deploy a preset wake-up module for breath wake-up in the low-power space of other processors, such as low-power memory space. The design of the low-power space allows specific tasks, such as processing audio data or sensor data, to continue processing while other parts of the system (such as the AP) are in sleep mode, thereby reducing the power consumption of the electronic device.

[0052] For example, the other processor can be a coprocessor, also known as an auxiliary processor. The coprocessor continues to operate regardless of whether the main processor (AP) is in sleep mode. For instance, the coprocessor could be an Audio Digital Signal Processor (ADSP), a System Control Processor (SCP), or a sensor hub chip. When the user performs voice wake-up, the other processor can wake up the AP to perform subsequent operations.

[0053] Taking ADSP as an example, all operations performed by ADSP can be implemented by the aforementioned other processors. This application's embodiments are merely illustrative using ADSP as an example and do not imply that this application is limited thereto. The inventors, in implementing these embodiments, also discovered that when the low-power memory space of the chip platform's ADSP is sufficiently large, all preset wake-up modules are deployed within the ADSP's low-power memory space. However, if the low-power memory space of the chip platform's ADSP is limited, it is impossible to deploy all preset wake-up modules within the ADSP's low-power memory space. Therefore, limited by the ADSP's low-power memory space, the breath-activated wake-up method is difficult to universalize.

[0054] Taking the main processor as an application processor (AP) as an example, the AP entering sleep mode can be understood as entering a low-power mode, which reduces energy consumption and extends the usage time of electronic devices. When the AP enters sleep mode, it includes, but is not limited to, the following situations: 1. The AP's clock frequency is reduced or stopped. The clock is the basis for the processor to execute instructions; reducing or stopping the clock frequency can reduce the processor's power consumption. 2. To match the reduced clock frequency, the AP's supply voltage is reduced accordingly. 3. Some circuits are shut down or suspended, such as the cache and graphics processor.

[0055] For electronic devices with indicator lights, when the AP enters sleep mode, the indicator lights may flash in a specific way or remain constantly on / off. For electronic devices with displays, when the AP enters sleep mode, the screen may turn off or display a specific sleep screen saver.

[0056] Taking the AP as the main processor as an example, all operations performed by the AP can be implemented through the main processor. The main processor can enter a sleep state or a low-power mode, which may include the AP entering a sleep state or a low-power mode as described above. This application embodiment is only used as an example of AP and does not mean that this application is limited to this.

[0057] This application provides a voice interaction method. At a first moment, the electronic device is lifted while its screen is off. The power consumption of the electronic device changes from a first value to a second value. Before the first moment, the screen is off and the device is not lifted, so the second value is greater than the first value. Between the first and second moments, the electronic device receives a first voice message. At the second moment, the screen lights up, and after the second moment, the power consumption is a third value, which is greater than the second value. Therefore, before the first moment, the screen is off and the main processor is in sleep mode, at which point the power consumption is the first value. After the device is lifted, the first voice message is received. By storing the first voice message in the non-low-power storage space of the device's secondary processor only after the device is lifted, the power consumption of the electronic device is reduced. From before the first moment until the device is detected being lifted, and then from the third moment until the user wakes up the main processor with the first voice message and the screen lights up, the power consumption of the electronic device gradually increases, changing from the first value to the second value, and then from the second value to the third value. This ensures low power consumption when the device is woken up by breath.

[0058] Specifically, by deploying the preset wake-up modules separately in the ADSP's storage space, the dependence of the preset wake-up modules on low-power memory is reduced, improving the universality of the breath-activated wake-up method. Optionally, the preset wake-up modules can be deployed separately in the ADSP's memory space, and different functional modules of the preset wake-up modules can be deployed separately in the ADSP's low-power memory space and non-low-power memory space, further reducing the preset wake-up modules' dependence on low-power memory space. For example, some functional modules of the preset wake-up modules are deployed in the ADSP's low-power memory space, while other functional modules are deployed in the ADSP's non-low-power memory space.

[0059] The ADSP operates some functional modules deployed in the low-power memory space to detect the motion state of the electronic device, i.e., whether the electronic device has been lifted. At this time, the power consumption of the electronic device is a first value. Upon receiving the detection result that the electronic device has been lifted, the ADSP operates another set of functional modules deployed in the non-low-power memory space to acquire voice (such as the first voice) for breath wake-up detection, confirming whether to wake up the AP (or voice interaction module) in a dormant state. At this time, the power consumption of the electronic device is a second value. If the AP (or voice interaction module) in a dormant state is woken up, the electronic device screen turns on, and the power consumption of the electronic device is a third value, which is greater than the second value. Therefore, the reliance of the preset wake-up module on the low-power memory space is reduced. Even if the low-power memory space of the ADSP is limited, the voice interaction method provided in this application embodiment can still achieve the wake-up of the voice interaction module by breath wake-up when the electronic device screen is off and the AP is in a dormant state.

[0060] The voice interaction method provided in this application can be applied to electronic devices, such as mobile phones, tablets, wearable devices, in-vehicle devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and home appliances. This application does not impose any restrictions on the specific type of electronic device. The electronic device can be a device running Android, iOS, Windows, or other operating systems. This application does not specifically limit the type of electronic device executing the voice interaction method provided in this application or the operating system running on that device.

[0061] The electronic device provided in this application embodiment has a voice interaction module. For example, the electronic device has installed a voice assistant application (such as YOYO) that provides voice interaction functions. The user wakes up the voice assistant to realize voice interaction between the user and the electronic device. Activating the voice assistant installed on the electronic device is equivalent to activating the voice interaction module of the electronic device.

[0062] The "voice assistant" mentioned in this application embodiment can also be referred to as a "smart assistant," "digital assistant," "virtual assistant," "intelligent automated assistant," or "automatic digital assistant," etc. A "voice assistant" can be understood as an information processing system that can recognize natural language input in the form of speech and / or text to infer the user's intent and perform corresponding actions based on the inferred user intent. This system can output responses to user input in audible (e.g., voice) and / or visual forms.

[0063] The electronic device provided in this application embodiment has a preset wake-up module for waking up the voice interaction module. For example, a voice assistant includes a voice wake-up module and a preset wake-up module. When the voice wake-up module is activated, the user can wake up the smart assistant using a wake-up word. The electronic device wakes up the smart assistant in response to receiving a specific wake-up word. When the preset wake-up module is activated, the user can wake up the smart assistant without using a wake-up word; the user can wake up the smart assistant using breath. The electronic device wakes up the smart assistant in response to being in a raised state or in response to a hand-raising event, receiving audio data sent by the user.

[0064] The preset wake-up module can also be called a "wake-up word-free wake-up module" or a "breath wake-up module," etc., and this application does not make a specific limitation on it.

[0065] The following uses a mobile phone as an electronic device and a smart assistant as a voice interaction module as an example to illustrate the voice interaction scenario provided in the embodiments of this application.

[0066] like Figure 1A As shown, the phone displays the home screen interface 11. The home screen interface 11 displays various types of application (App) icons, such as the phone icon, contacts icon, settings icon 101, and messages icon.

[0067] The mobile phone receives input from the user on the settings icon 101 and, in response to the input on the settings icon 101, displays the settings interface 12 as shown in 1B.

[0068] like Figure 1BAs shown, the settings interface 12 may include management bars for multiple services, such as a WLAN management bar, Bluetooth management bar, mobile network management bar, notification and status bar, desktop and wallpaper management bar, battery management bar, storage management bar, security management bar, accessibility management bar, and smart assistant management bar 102. In practical applications, the settings interface 12 may also include more or fewer types of management bars than shown in the figure, and this application does not limit this.

[0069] The mobile phone receives input from the user on the smart assistant management bar 102 and, in response to the input on the smart assistant management bar 102, displays the smart assistant interface 13 as shown in 1C.

[0070] like Figure 1C As shown, the smart assistant interface 13 may include a YOYO suggestion control, a negative one screen control, a smart vision control, and a smart voice control 103. The smart voice control 103 is used to provide wake-up settings for voice interaction services.

[0071] The mobile phone receives input from the user to the smart voice control 103 and, in response to the input, displays the smart voice interface 14 as shown in Figure 1D.

[0072] like Figure 1D As shown, the intelligent voice interface 14 may include a voice wake-up control, a breath wake-up control 104, a button wake-up control, a desktop shortcut control, a broadcast tone control, and a volume control, etc.

[0073] To activate the voice wake-up module, the user can click the voice wake-up control. To activate the preset wake-up module, the user can click the breath wake-up control 104.

[0074] The mobile phone receives input from the user to the breath wake-up control 104, and in response to the input, can display something like... Figure 1E The aura awakening interface 15 shown is the second interface.

[0075] The breath wake-up interface 15 may include instructions for breath wake-up (i.e., the preset wake-up module), operation prompts, and a switch control 105 (i.e., the first control). The operation prompts may use animations to demonstrate how the user can wake up the smart assistant using the breath wake-up function. The operation prompts may also include text prompts, such as "Natural dialogue, instant response, no need to say 'Hello YOYO' wake-up word." The breath wake-up instructions describe how to use the breath wake-up method. These instructions may include text prompts such as: "Raise your phone, bring the bottom of the phone close to your mouth (within 5cm), and speak a voice command directly into the bottom microphone, such as 'How's the weather today?'" Thus, when the preset wake-up module is running, the user can wake up the smart assistant by directly outputting a voice command without first outputting a wake-up word.

[0076] The mobile phone receives user input to the switch control 105 and, in response to the input, activates the phone's preset wake-up module. Before receiving input, the switch control 105 is displayed in its first state (e.g., ...). Figure 1E As shown), the switch control 105, displayed in its first state, indicates that the phone has not activated the breath wake-up function, and the phone cannot wake up the voice interaction module via breath wake-up. After the switch control 105 receives an input operation, the switch control 105 is displayed in its second state (as shown). Figure 1G As shown, the switch control 105, displayed in the second state, indicates that the phone has activated the breath wake-up function, i.e., the wake-up word-free function. The phone can wake up the voice interaction module through breath wake-up (or wake-up word-free method). It is understood that the above input operation can be a click, a long press, etc., and this application does not specifically limit it.

[0077] It's understandable that, besides the user-activated preset wake-up module mentioned above, it can also be activated through system default settings, meaning the preset wake-up module is activated by default in the phone's factory settings. If a user wants to disable the preset wake-up module, they can do so through methods such as... Figures 1A to 1E The operation shown indicates that the preset wake-up module is disabled.

[0078] It is understood that the above-described method of enabling the preset wake-up module is merely an example to illustrate the implementation of enabling the preset wake-up module in the embodiments of this application. Other methods of enabling the preset wake-up module and the different technical solutions formed in the embodiments of this application are all within the protection scope of this application.

[0079] After the user activates the phone's preset wake-up module, the user can wake up the smart assistant using breath. For example, when the phone screen is off and the phone's application processor (AP) is in sleep mode, the smart assistant can be woken up by breath. Another example is waking up the smart assistant by breath when the phone screen is on.

[0080] For example, after the user activates the phone's preset wake-up module, the user presses the phone's power button to turn off the screen, and the AP enters sleep mode. Figure 1F As shown, microphone 1 and microphone 2 are located at the top of the phone. When the screen is off, the user can lift the phone, bring the bottom of the phone close to their mouth, and speak into microphone 2. Both microphone 1 and microphone 2 can collect the audio data corresponding to the user's speech. Sensors within the phone, such as accelerometers, gyroscopes, and inertial measurement units (IMUs), collect the phone's motion data.

[0081] The distance between the mobile phone and the user's mouth can be maintained at 0-5cm, which makes it easy for the phone's microphones 1 and 2 to accurately collect the user's audio data.

[0082] Among them, by Figure 1F The gesture state indicated by section A in the middle frame, to Figure 1F As shown in section B of the diagram, the user can lift the phone by raising their wrist. The phone's internal sensors can collect motion data, including acceleration, tilt, impact, vibration, and rotation data. For example... Figure 1F As shown, the phone is lifted up, the phone is currently in a hand-held lifted state (i.e., lifted state) or the phone has a hand-lifting event. The hand-held lifted state can be understood as the user holding the phone in their hand.

[0083] Understandable. Figure 1F This is merely one example of a phone being lifted. A user can also lift a stationary phone, bring the bottom of the phone close to their mouth, thus lifting the phone, placing it in a lifted state, or triggering a hand-raising event. Electronic devices can be lifted, placed in a lifted state, or trigger hand-raising events in other ways, and this application does not specifically limit this.

[0084] It's understandable that the "raised state" refers to a state within a specific timeframe; the phone is in a raised state for the duration it's being raised. For example, if the phone is placed in a location and the user picks it up, the phone is in the process of being raised, and this is either a raised-hand event or a hand-raising event occurs. Another example is when the user holds the phone and moves it to input voice; in this case, the phone is in a raised state or a hand-raising event occurs. Moving the phone to input voice could be, for example, moving the phone to the user's mouth.

[0085] like Figure 1F As shown, after the user raises their phone, they speak the voice message "How's the weather today?" With the phone raised, both microphones 1 and 2 can pick up the voice message "How's the weather today?" (as shown in the first voice message).

[0086] It's understandable that, given the different positions of microphones 1 and 2 on the phone, the distances between microphone 1 and the sound source (user) are different, and the distances between microphone 1 and microphone 2 are also different. Therefore, the signal strength of the speech received by microphones 1 and 2 may differ, potentially resulting in an energy difference between the received speech signals. When the user speaks towards the top of the phone, the speech energy captured by the top microphone 1 will be higher than that captured by the bottom microphone 2. Similarly, when the user speaks towards the bottom of the phone, the speech energy captured by the bottom microphone 2 will be higher than that captured by the top microphone 1.

[0087] The ADSP determines whether a breath-activated wake-up event has been triggered. When a breath-activated wake-up event is detected, the ADSP wakes up the AP, and the AP starts or resumes the smart assistant. When a breath-activated wake-up event is not triggered, the ADSP neither wakes up the AP nor starts or resumes the smart assistant.

[0088] For example, when the phone is raised, it collects the first voice corresponding to the user's speech through microphones 1 and 2, as well as motion data collected by sensors, and then sends the collected first voice and motion data to the ADSP. The ADSP detects the first voice and motion data through a preset wake-up module to determine whether a breath wake-up event has been triggered.

[0089] like Figure 1G As shown, when a breath-activated wake-up event is detected based on the first voice and / or motion data, the ADSP wakes up the AP, the phone screen lights up, and the wake-up interface 16 of the smart assistant is displayed. The wake-up interface 16 includes text content 106 and a prompt control 107. The text content 106 is related to the audio data entered by the user when waking up the smart assistant. For example... Figure 1FAs shown, the user inputs the audio data "How's the weather today?", and the text data displayed in text content 106 is also "How's the weather today?". The prompt control 107 is used to remind the user that the smart assistant has been activated. The user can input an action (such as clicking) on ​​the prompt control 107 to trigger the smart assistant to end voice recognition.

[0090] In some embodiments, the wake-up interface 16 may also include interface content before the smart assistant is woken up. For example... Figure 1G As shown, the interface content before the smart assistant is awakened includes the content of the breath awakening interface 15.

[0091] It's understandable that when the audio data is "How's the weather today?", it indicates that the user wants to know today's weather. After the smart assistant determines that voice recognition is complete based on the audio data, the phone can display something like this: Figure 1H The voice interaction interface 17 is shown. The voice interaction interface 17 includes voice command text 108. Voice command text 108 includes text data from the received audio data (i.e., voice commands) converted into text by the smart assistant. For example... Figure 1H As shown, the voice command text 108 displays "How's the weather today?"

[0092] In some embodiments, the voice interaction interface 17 may further include response information 109. Response information 109 displays the processing result of the voice command "How's the weather today?" in the voice command text 108. Figure 1H As shown, when the voice command is "How's the weather today?", the processing result displayed in response information 109 includes "Shenzhen, mostly cloudy, probability of rain: 50%, current temperature 32℃, high temperature 32℃, low temperature 27℃". Response information 109 can also include weather information for various time periods.

[0093] The first interface may include a wake-up interface 16 and / or a voice interaction interface 17. The first interface includes the content of the voice (such as the first voice) received when the phone is lifted and / or after the phone is lifted. Taking the content of the first voice as an example, the content of the first voice in the first interface may include, but is not limited to, the following: (1) The first interface may include text, pictures, videos or other media related to the content of the first voice. For example, if the user says "How is the weather today?", the first interface may display the text "How is the weather today?" of the first voice. (2) The first interface may include the processing result of the instruction in the first voice, which may be the search result of the keyword in the first voice. For example, if the user says "How is the weather today?", the first interface may display the current weather information. For another example, if the user says a query, the first interface will display the relevant search results or information. (3) The first interface may include a series of operation options or instructions related to the first voice. For example, if the user says "Set alarm", the first interface will display the specific time and method of setting the alarm. (4) The first interface may include feedback or confirmation that the user's voice instruction has been received, and display a summary or execution status of the instruction. For example, if a user says "turn off the music", the first screen can display "Instruction received, music turned off".

[0094] The triggering time for the aforementioned breath awakening event includes the following situations, and it can be triggered in any of the following ways; this application does not specifically limit this:

[0095] Scenario 1: The phone is detected to be lifted, and the second signal strength of the first voice collected by microphone 2 is detected to be greater than the first signal strength of the first voice collected by microphone 1.

[0096] After the phone activates the preset wake-up module, the user turns off the phone screen. For example... Figure 1F As shown, the phone is off when it is lifted. During the time between the phone being lifted and the screen turning on, the phone receives the first voice message, "How's the weather today?". When the phone receives the first voice message, the third distance between the sound source of the first voice message (e.g., the user) and microphone 2 (i.e., the second microphone) is less than the fourth distance between the sound source of the first voice message (e.g., the user) and microphone 1 (i.e., the first microphone). Therefore, it can be detected that the second signal strength of the first voice message collected by microphone 2 is greater than the first signal strength of the first voice message collected by microphone 1. Figure 1F In the process, the system detects that the phone has been lifted and that the second signal strength of the first voice signal collected by microphone 2 is greater than the first signal strength of the first voice signal collected by microphone 1, triggering a breath wake-up event, and then... Figure 1G and Figure 1HAs shown, the phone screen lights up. The phone can also display an interface related to the first voice message, such as... Figure 1G and Figure 1H As shown.

[0097] Optionally, if the difference between the second signal strength of the first speech acquired by microphone 2 and the first signal strength of the first speech acquired by microphone 1 (i.e., the first difference) is greater than a first threshold, a breath wake-up event is triggered.

[0098] It is understandable that the first threshold can be set according to the actual situation, and this application does not make specific restrictions on it.

[0099] In scenario 1, the first speech may include a human voice, i.e., the sound of a person speaking. Optionally, the first speech in scenario 1 may include, but is not limited to, a human voice.

[0100] Scenario 2: The phone is detected to be lifted, and the first voice is detected to include a human voice.

[0101] After the phone activates the preset wake-up module, the user turns off the phone screen. For example... Figure 1F As shown, the phone is initially off, then it is lifted up. During the time between the phone being lifted and the screen turning on, the phone receives the first voice message, "How's the weather today?" The phone detects that the first voice message, "How's the weather today?", is a human voice, triggering a breath-activated wake-up event, and then... Figure 1G and Figure 1H As shown, the phone screen lights up. The phone can also display an interface related to the first voice message, such as... Figure 1G and Figure 1H As shown.

[0102] The difference between scenario 2 and scenario 1 is that scenario 2 does not limit the positional relationship between the sound source of the first voice and the mobile phone microphone. For example, when the mobile phone receives the first voice, if the third distance between the sound source of the first voice (such as the user) and microphone 2 (i.e., the second microphone) is greater than or equal to the fourth distance between the sound source of the first voice (such as the user) and microphone 1 (i.e., the first microphone), then the second signal strength of the first voice collected by microphone 2 is detected to be less than or equal to the first signal strength of the first voice collected by microphone 1. Based on the determination that the first voice is a human voice, a breath wake-up event is also triggered.

[0103] In some embodiments, the mobile phone may pre-store the user's voiceprint, detect that the first voice "How's the weather today?" matches the pre-stored user's voiceprint, and determine that the first voice is a human voice.

[0104] It's understandable that the first speech signal can be detected as a human voice in other ways, such as using machine learning algorithms (like support vector machines and neural networks) to determine if it's a human voice. Another example is identifying whether a human voice is present in a sound signal by analyzing the sound characteristics (such as frequency and amplitude) of the sound signal captured by the microphone.

[0105] Scenario 3: The phone is detected to be raised, the first voice is human voice, the second signal strength of the first voice collected by microphone 2 is detected to be greater than the first signal strength of the first voice collected by microphone 1, and the second signal strength is greater than the second threshold.

[0106] Optionally, if the difference between the second signal strength of the first speech acquired by microphone 2 and the first signal strength of the first speech acquired by microphone 1 (i.e., the first difference) is greater than a first threshold, the second signal strength is greater than a second threshold, and the first speech includes human voice, a breath wake-up event is triggered.

[0107] It is understandable that the second threshold can be set according to the actual situation, and this application does not make specific restrictions on it.

[0108] It should be noted that the above triggering scenarios are for reference only, to illustrate the implementation of triggering the breath awakening event in the embodiments of this application. Other ways of triggering the breath awakening event and the different technical solutions formed in this embodiment are all within the protection scope of this application.

[0109] When the ADSP determines that the smart assistant needs to be activated, it generates a wake-up signal and transmits it to the AP. For example, the ADSP detects a breath-activated wake-up event and generates a wake-up signal. Alternatively, the ADSP detects a wake word and generates a wake-up signal. The ADSP can send the wake-up signal to the AP through internal system communication interfaces such as I2C, SPI, or a dedicated wake-up line.

[0110] If the AP was previously in a low-power mode, such as sleep mode, it will resume normal operation during the wake-up process. For example, upon receiving a wake-up signal, the AP will wake up or configure necessary system resources, including but not limited to: adjusting power management policies, increasing clock frequency, and enabling necessary hardware interfaces. For instance, after the AP's Power Management Unit (PMU) recognizes the wake-up signal, the PMU changes the AP's power state to provide the required voltage and current, such as activating the power converter, adjusting the voltage level, or increasing the supply current. Another example is that the AP's Clock Management Unit (CMU) adjusts the clock frequency.

[0111] After receiving the wake-up signal, the AP checks the status of the smart assistant and takes appropriate action based on the status.

[0112] If the smart assistant is not already running, the AP launches the smart assistant application. Launching the smart assistant application by the AP may include, but is not limited to: initializing resources, loading the application, launching the application, preparing the user interface, and initializing services and components. For example, initializing resources may include, but is not limited to: the AP allocating necessary resources to the smart assistant application, including CPU time slices, memory, and possible I / O resources. For example, loading the application may include, but is not limited to: the AP loading the application's code and resources from storage into memory. For example, launching the application may include, but is not limited to: the AP launching the entry point of the smart assistant application, such as the main Activity or a specific launch Service. For example, preparing the user interface may include, but is not limited to: the AP preparing and displaying the user interface so that the user can see the application's status and feedback.

[0113] If the smart assistant is already running in the background but is in a sleep or low-power state, the AP wakes it up and restores it to normal operation. This process can include, but is not limited to: waking the smart assistant, allocating computing resources, restoring services, and updating the user interface. For example, waking the smart assistant can include, but is not limited to: the operating system changing the smart assistant's state from sleep or low-power to normal operation. Allocating computing resources can include, but is not limited to: the AP reallocating necessary computing resources, such as CPU and memory, to ensure the smart assistant can function normally. Restoring services can include, but is not limited to: the AP restarting services that stopped due to low power consumption. Updating the user interface can include, but is not limited to: if the smart assistant has a user interface, the AP ensuring the interface displays correctly after waking up and updating any pending notifications or information. The AP restores interactive functions between the user and the smart assistant, such as clicking, swiping, and voice input, so that the user can continue to use the smart assistant for various operations.

[0114] If some functions of the smart assistant are running in the background, the AP restores the full functionality of the smart assistant, including but not limited to: user interface display and background services. For example, restoring the main interface display may include, but is not limited to: the smart assistant's main interface (such as the negative one screen, sidebar, etc.) being displayed normally on the screen, restoring interactive functions between the user and the smart assistant, such as clicking, swiping, and voice input, and restoring data synchronization, such as user preferences, settings, and history. For example, restoring various services running in the background of the smart assistant may include, but is not limited to: notification reminders, health monitoring, and security protection. It is understood that the above explanation uses the voice interaction module as an example of the smart assistant. After the ADSP wakes up the AP, the AP can refer to the above method to wake up the voice interaction module. If the voice interaction module has not yet been started, the AP starts the voice interaction module. The AP starting the voice interaction module may include, but is not limited to: initializing resources, loading applications, starting applications, preparing the user interface, and initializing services and components; details can be found above. If the voice interaction module is already running in the background, but is in a sleep or low-power state, the AP wakes up the voice interaction module and restores it to normal working status; details can be found above. If some functions of the voice interaction module are running in the background, the AP will restore the full functionality of the voice interaction module, including but not limited to: user interface display and background services.

[0115] It should be understood that the above are illustrative examples of application scenarios and do not limit the application scenarios of this application in any way.

[0116] It is understood that the text, controls and other visible interface elements displayed in the above interface are merely examples, and this application does not limit the display position, display style, display size and other parameters of the interface.

[0117] Taking a mobile phone as an example, the hardware structure of the electronic device provided in the embodiments of this application will be introduced.

[0118] like Figure 2 As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0119] The aforementioned sensor module 180 may include a gyroscope sensor 180A and an accelerometer sensor 180B. The gyroscope sensor 180A may be a three-axis gyroscope, used to track the state changes of the electronic device 100 in six directions. The accelerometer sensor 180B is used to detect the speed, direction, and displacement of the electronic device 100.

[0120] In some embodiments, sensor module 180 may include an inertial measurement unit (IMU). An inertial measurement unit can also be referred to as an inertial sensing sensor. An IMU is a device that integrates multiple inertial sensors for measuring and tracking the acceleration and angular velocity of an object. Typically, an IMU includes three gyroscopes (such as gyroscope sensor 180A) for measuring angular velocity and three accelerometers (such as accelerometer sensor 180B) for measuring acceleration. These sensors are typically combined to provide information about the attitude, angular velocity, and acceleration of electronic device 100 in three-dimensional space.

[0121] It is understood that the pose sensor of the electronic device 100 may include, but is not limited to: gyroscope sensor 180A, accelerometer sensor 180B, inertial detection sensor and inertial measurement unit.

[0122] In some embodiments, the sensor module 180 may also include other sensors, such as a proximity light sensor.

[0123] Processor 110 may include one or more processing units, such as: an application processor (AP), an audio digital signal processor, a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. An application processor is a general-purpose microprocessor that can perform a wide range of tasks, including but not limited to: running an operating system, processing applications, and managing device resources. Application processors are typically integrated into a system-on-a-chip (SOC), which may also include a central processing unit (CPU), a GPU, a memory controller, and other necessary hardware components. A digital signal processor is a microprocessor dedicated to digital signal processing; it uses digital technology to process signals. A DSP consists of large-scale or very-large-scale integrated circuit chips used to perform various complex signal processing tasks.

[0124] An audio digital signal processor (DSP) is a device that uses digital technology to process and optimize sound; it is a processor specifically designed for processing digital audio signals. It is commonly used to improve sound quality, eliminate noise, equalize volume, enhance bass and treble, and achieve other audio effects. While a DSP is a general-purpose digital signal processor, an ADSP is a dedicated processor focused solely on audio signal processing.

[0125] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0126] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system. In some embodiments, the processor 110 may include one or more interfaces.

[0127] The charging management module 140 receives charging input from a charger, which can be a wireless charger or a wired charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0128] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0129] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 360, modem processor and baseband processor, etc.

[0130] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0131] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0132] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0133] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor.

[0134] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. For example, the display screen 194 may be a touchscreen.

[0135] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0136] Electronic device 100 can implement audio functions through audio module 370, speaker 370A, receiver 370B, microphone 370C, headphone jack 370D, and application processor. For example, microphone 370C can collect user voice information, and speaker 370A can play feedback results on the user's voice information.

[0137] Microphone 370C may include one or more microphones. When the electronic device 100 includes multiple microphones, these multiple microphones may be respectively positioned at different locations within the electronic device, so that the signal strength of the voice received by the multiple microphones varies. Figure 1F As shown, the mobile phone includes a microphone 1 located at the top of the phone and a microphone 2 located at the bottom of the phone.

[0138] It is understood that the audio acquisition device of electronic device 100 may include, but is not limited to, a microphone.

[0139] A touch sensor, also known as a "touch panel," can be located on the display screen 194. The touch sensor and the display screen 194 together form a touch screen, also known as a "touchscreen." In other embodiments, the touch sensor may also be located on the surface of the electronic device 100, in a different position than the display screen 194.

[0140] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0141] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0142] The voice interaction methods provided in this application can all be implemented in the electronic device 100 having the above-described hardware structure.

[0143] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100.

[0144] In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0145] In this embodiment of the application, the electronic device 100 can implement sound processing and service implementation by setting up a sound processing device. For example... Figure 3 As shown, the audio processing device in electronic device 100 for implementing audio service processing may include a system-on-chip (SOC) and a codec, with the SOC and codec connected. The codec performs processing on the audio signal acquired by the audio acquisition device (e.g., a microphone), such as encoding and decoding, and sends the processed audio signal to the SOC. The SOC then performs corresponding service processing based on the audio signal. The codec may be, for example, an analog-to-digital (AD) and / or a digital-to-analog (DA) codec.

[0146] In this embodiment, the SOC may include a first processor and a second processor. The first processor and the second processor can be connected via a bus, such as an on-chip bus (OCB). The voice interaction module of the electronic device can be deployed on the first processor, and the preset wake-up module of the electronic device can be deployed on the second processor.

[0147] It is understood that the first processor and the second processor may adopt, but are not limited to, the following connection methods: high-speed serial connection (Peripheral Component Interconnect Express, PCIe), dedicated communication interface, serial communication interface, such as Serial Peripheral Interface (SPI), Inter-Integrated Circuit (I2C), and Universal Asynchronous Receiver / Transmitter (UART).

[0148] It is understood that the communication methods between the first processor and the second processor may include, but are not limited to: shared memory, direct memory access (DMA) data transfer communication.

[0149] It is understood that this application does not specifically limit the connection method and communication method between the first processor and the second processor.

[0150] like Figure 3 As shown, the first processor can be the main processor. The main processor can be the main computing unit in an electronic device, responsible for executing the operating system, running applications, and processing data. The main processor can be the most powerful and versatile processor in the electronic device.

[0151] For example, the first processor may include, but is not limited to, the following types: application processor, central processing unit, and multimedia application processor (MAP).

[0152] It is understood that, taking the first processor as an example above, all operations performed by the AP can be implemented through the first processor. The situation where the first processor enters a sleep state or a low-power mode can include the situation where the AP enters a sleep state or a low-power mode. The embodiments of this application are only illustrated by taking the AP as an example and do not mean that this application is limited to this.

[0153] The second processor can be an auxiliary processor. An auxiliary processor, also known as a coprocessor, is a processor in an electronic device that assists the main processor in performing specific tasks. An auxiliary processor can focus on specific functions or optimize specific types of computational tasks; for example, it could focus on waking a voice interaction module via breath.

[0154] Exemplary examples show that the second processor may include, but is not limited to, the following types: Digital Signal Processor (DSP), Audio Digital Signal Processor, Field-Programmable Gate Array (FPGA), and Application-Specific Integrated Circuit (ASIC). In some embodiments, the second processor may be a System Control Processor (SCP). In other embodiments, the second processor may be a Sensor Hub. A Sensor Hub is a microcontroller / auxiliary processor that helps acquire, integrate, and process data from different sensors. In some embodiments, the second processor may also be a Sensor Hub chip.

[0155] It is understood that the processor types of the first and second processors can be selected according to the actual situation, and this application does not make specific restrictions on this.

[0156] In some embodiments, the sound processing apparatus may further include sensors mounted on the audio digital signal processor, including but not limited to: accelerometers and gyroscopes. In some embodiments, the accelerometer and gyroscope are integrated in an inertial measurement unit, and the sound processing apparatus may further include an inertial measurement unit mounted on the audio digital signal processor.

[0157] The following describes the structure of the electronic device provided in the embodiments of this application, taking the first processor as the application processor (AP) and the second processor as the audio digital signal processor (ADSP). It should be noted that in the embodiments of this application, the operations performed by the AP can be implemented by the first processor, and the operations performed by the ADSP can be implemented by the second processor.

[0158] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0159] like Figure 4 As shown, the layered architecture of the application processor (AP) divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system may include an application layer, a framework layer, and a hardware abstraction layer (HAL).

[0160] The application layer can include applications (APPs). The application layer includes all applications installed on electronic devices, which may include voice-interactive applications, such as smart assistants.

[0161] A smart assistant provides voice interaction functionality. It can parse user voice commands and perform related operations accordingly, thus enabling voice interaction between the electronic device and the user.

[0162] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer includes some predefined functions.

[0163] The framework layer may include an audio trigger module (Sound Trigger, ST), an audio policy service module (AudioPolicy Service), an audio recording module, etc.

[0164] The audio triggering module in the framework layer is used to control the launch of voice interaction applications in the application layer.

[0165] The audio strategy service module is used to establish a voice recognition channel with voice interaction applications.

[0166] The audio recording module is used for audio recording. In some embodiments, the audio recording module can be implemented as AudioRecord, which is an API class provided by the Android framework layer for audio recording.

[0167] In some embodiments, the framework layer may also include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0168] The Hardware Abstraction Layer (HAL) is the layer between hardware and software. The HAL can include modules such as audio triggering modules. The audio triggering module in the HAL can also be called an audio trigger driver (ST-HAL).

[0169] The audio triggering module in the Hardware Abstraction Layer (HAL) is responsible for audio triggering functionality. It enables applications to listen for certain sound events, such as triggering breath-to-wake notifications, in a low-power and privacy-sensitive manner. The HAL audio triggering module handles hardware recognition of breath-to-wake notifications and other sounds.

[0170] The hardware abstraction layer can contain multiple drivers used to drive the hardware.

[0171] It should be noted that the application layer, framework layer, and hardware abstraction layer may also include other content, and no specific restrictions are made here.

[0172] It should be noted that the embodiments of this application are only illustrated using the Android system as an example. In other operating systems (such as Windows system, iOS system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of this application, the solution of this application can also be implemented.

[0173] also, Figure 4The diagram also illustrates an audio digital signal processor (ADSP) connected to an application processor (AP), a microphone connected to the ADSP, and sensors. The microphone connected to the ADSP may include a first microphone and a second microphone, which may be positioned at different locations on the electronic device, such as the top and bottom. Devices connected to the ADSP may include an accelerometer and a gyroscope. In some embodiments, the accelerometer and gyroscope may be integrated into an inertial measurement unit (IMU), and the IMU may be connected to the ADSP. Devices connected to the ADSP may include, but are not limited to, gyroscope sensors and / or accelerometer sensors and / or inertial detection units.

[0174] In the embodiments of this application, the ADSP may include a low-power space and a non-low-power space. The low-power space and the non-low-power space can be understood as storage spaces divided according to power optimization requirements.

[0175] Optionally, the low-power space can be a low-power memory space, and the non-low-power space can be a non-low-power memory space. Low-power and non-low-power memory spaces can be understood as different memory regions divided according to power optimization requirements. The low-power memory space uses power optimization techniques to reduce power consumption, while the non-low-power memory space focuses more on performance improvement, but may have higher power consumption.

[0176] The following explanation uses ADSP, which includes low-power memory space and non-low-power memory space, as an example. All operations performed in the low-power memory space can be implemented through the aforementioned low-power space, and all operations performed in the non-low-power memory space can be implemented through the aforementioned non-low-power space. The implementation of the low-power space can be referred to the following low-power memory space, and the implementation of the non-low-power space can be referred to the following non-low-power memory space.

[0177] In this application embodiment, the low-power memory space of an ADSP generally refers to a memory region designed to operate in the ADSP's low-power mode, which can remain active while certain parts of the processor enter a low-power state. The design of the ADSP's low-power memory space allows specific tasks, such as processing audio data or sensor data, to continue processing while other parts of the system (such as the AP) are asleep, thereby reducing overall power consumption.

[0178] In the embodiments of this application, the non-low-power memory space of the ADSP can be a normal memory region that provides data storage and processing functions when the system is active, but may not remain active when the ADSP enters low-power mode.

[0179] In this embodiment, the non-low-power memory space of the ADSP can be used for services that need to continue running after the AP (application processor) goes to sleep, such as voice wake-up, sensor data processing, step counting, etc.

[0180] In this embodiment, the low-power memory space of the ADSP can be the CPU's L2 cache. The L2 cache provides fast data access, and it can be implemented using low-power memory to reduce power consumption.

[0181] In this embodiment, the low-power memory space of the ADSP can be specifically implemented using low-power memory (i.e., a second memory). Low-power memory has low-power characteristics and is suitable for specific services that need to continue running when the AP is in sleep mode. Low-power memory can include, but is not limited to, Static Random-Access Memory (SRAM). As long as SRAM is powered on, the data stored within it can be constantly maintained. SRAM does not require a refresh circuit to retain its internally stored data, which makes its power consumption relatively low during operation. Due to the static storage characteristics of SRAM, its power consumption is low when no read / write operations are performed, making it very suitable for low-power memory spaces with strict power requirements. Using SRAM in the ADSP's low-power memory space can ensure stable data storage while reducing overall power consumption.

[0182] In this embodiment, the low-power memory space of the Audio Digital Signal Processor (ADSP) includes an audio region and a sensing region. The audio region and sensing region can be understood as different working areas divided according to business needs. The audio region is mainly responsible for audio-related tasks, such as processing tasks directly related to audio signals. The audio region can run algorithms related to audio processing. The sensing region is mainly responsible for sensor data-related tasks, such as the acquisition and processing of sensor data. The sensing region can run algorithms related to sensor data processing.

[0183] In this embodiment, the low-power memory space of the ADSP is divided into different physical regions; the sensor space (i.e., the sensing area) and the audio space (i.e., the audio area) are separate. The sensing area can be implemented using a sensor low-power island (SLPI), and the audio area can be implemented using audio low-power island technology. Both the sensor low-power island and the audio low-power island can be implemented using low-power memory such as SRAM as the memory.

[0184] In ADSP, SLPI can be understood as a hardware architecture specifically designed for processing sensor data in low-power mode. The SLPI architecture is used to implement low-power memory spaces related to sensor data, such as the sensing area. Specifically, this can involve integrating modules that require continuous operation, such as drivers and algorithms for specific services (e.g., sensors), into the SLPI space. This allows the AP to be powered solely by the SLPI and continue operating when the AP is in sleep mode. In this way, the AP can enter a deep sleep state to reduce power consumption, while the SLPI space handles tasks that require continuous operation. Therefore, the AP can enter a sleep state when not needed, thereby reducing the overall system power consumption.

[0185] In the SLPI architecture, sensors (such as IMU, gyroscope sensor 180A, and accelerometer sensor 180B), microphones, etc., can be electrically connected to the ADSP so that the ADSP can process the data collected by the sensors. When the AP goes into sleep mode to reduce power consumption, the ADSP can process the sensor data, thereby achieving lower power consumption while ensuring that some functions are not affected by the AP going into sleep mode.

[0186] For example, the aforementioned sensors and microphones are mounted on the ADSP, such as being integrated on or connected to the same circuit board as the ADSP. In some embodiments, the drivers for the sensors and microphones run on the ADSP. In other embodiments, the drivers for the sensors and microphones run on an application processor (AP). The AP interacts with the sensors and microphones through a communication interface with the ADSP (such as I2C, SPI, UART, etc.).

[0187] In ADSP, the audio low-power island is a functional area specifically designed for audio processing, used to process audio data in a low-power mode. It can continue operating while the main processor is in sleep mode, thus reducing the overall system power consumption. This design allows audio data to be processed without CPU intervention, such as in scenarios like waking up a voice interaction module, where voice data can be directly transmitted without waking the CPU.

[0188] Audio low-power islands typically leverage system hardware features, such as using low-power SRAM as memory. Because SRAM doesn't require frequent refreshing like Dynamic Random Access Memory (DRAM), it consumes less power. This space can be used to store service data that needs to continue running after the AP goes to sleep, such as sensor data processing, step counting, and voice wake-up.

[0189] In this embodiment, by placing the modules that need to run continuously within the preset wake-up module in the low-power memory space of the ADSP, it can be ensured that these modules can still operate normally when the AP is in sleep mode without significantly affecting the overall power consumption. The partitioning of the ADSP's low-power memory space allows audio and sensor data processing to still be handled by a dedicated DSP (such as the ADSP) when the main processor is inactive, thereby optimizing power consumption. In this way, the ADSP can continue to provide high-quality audio processing functions while maintaining low power consumption.

[0190] Optionally, the parts of the preset wake-up module that need to run continuously can be placed in the low-power memory space of the ADSP, that is, the parts of the code that need to run continuously in the preset wake-up module can be placed in the low-power memory space of the ADSP.

[0191] It is understood that, taking the second memory as a low-power SRAM as an example, all the functions of the low-power SRAM can be implemented through the second memory. The embodiments of this application are only illustrated by taking the low-power SRAM as an example, and do not mean that this application is limited to this.

[0192] In this embodiment, the non-low-power memory space of the ADSP can be implemented using a non-low-power memory (i.e., the first memory). The non-low-power memory can include, but is not limited to, DRAM, Double Data Rate Synchronous Dynamic Random Access Memory (DRAM), or other types of memory.

[0193] DRAM is a type of dynamic random access memory that requires periodic refreshing to maintain data integrity. In the non-low-power memory space of ADSPs, where power consumption is not a primary consideration, highly integrated and relatively inexpensive DRAM can be used as the memory. DRAM's large capacity and relatively low power consumption make it an ideal choice for non-low-power memory space storage.

[0194] DDR is a synchronous dynamic random access memory with double the data transfer rate. Employing double data rate technology, it can transfer data on both the rising and falling edges of each clock cycle, thus achieving high-speed data transfer. DDR has a large storage capacity and supports various capacity specifications to meet the large data storage needs of ADSPs. DDR provides large-capacity, high-speed, and low-power data storage and transfer support for non-low-power memory spaces.

[0195] It is understood that the low-power memory space and non-low-power memory space in an ADSP can also be implemented using other memory types. For example, the low-power memory space can also be implemented using Low Power Double Data Rate Synchronous Dynamic Random Access Memory (LPDDR). Depending on the specific requirements and power consumption limitations of the ADSP, an appropriate memory type can be selected to implement the functions of the low-power memory space and non-low-power memory space; this application does not specifically limit this in its embodiments.

[0196] In this embodiment, low-power mode is an operating mode designed by the ADSP to reduce power consumption. When the ADSP is in low-power mode, its low-power memory space can be used to reduce overall power consumption. In non-low-power mode, the ADSP's non-low-power memory space can be used, thereby achieving optimal performance and bandwidth.

[0197] Specifically, when executing the voice interaction method provided in this application embodiment, if the ADSP is in low-power mode, it can use the ADSP's low-power memory space instead of its low-power memory space. That is, the low-power memory in the low-power memory space is active, while the non-low-power memory in the non-low-power memory space is in a low-power state. Conversely, if the ADSP is in non-low-power mode, it can use the ADSP's non-low-power memory space instead of its low-power memory space. That is, the low-power memory in the non-low-power memory space is active.

[0198] In this embodiment, the ADSP's exit from and entry into low-power mode can be understood as a switching of the operating mode of non-low-power memory (such as DDR) in the non-low-power memory space. Taking non-low-power memory DDR as an example, when the ADSP enters low-power mode, the DDR is in a low-power state. When the DDR is in a low-power state, power consumption can be reduced by lowering the operating frequency or entering low-power mode. When the ADSP exits low-power mode, the DDR enters the operating state. When the DDR is in the operating state, it may run at full speed or operate according to the conventional memory access mode to meet the ADSP's requirements for high performance and response speed.

[0199] In some embodiments, when the DDR is in a low-power state, the DDR operates with very little current; the DDR may only need to maintain data integrity but cannot read or write data. When the DDR is in a normal state, the DDR operates with a large current. The operating current of the DDR in a low-power state is less than the operating current of the DDR in a normal state.

[0200] In some embodiments, when the DDR is in a low-power state, the DDR operates at a very low voltage, and the DDR may only need to maintain data integrity but cannot read or write data. When the DDR is in a normal state, the DDR operates at a very high voltage. The operating voltage of the DDR in a low-power state is lower than the operating voltage of the DDR in a normal state.

[0201] It is understood that the above example of the first memory being a non-low-power memory DDR or DRAM can be used to illustrate the functions of the non-low-power memory DDR or DRAM. The embodiments of this application are only illustrated by taking non-low-power memory DDR or DRAM as examples and do not mean that this application is limited to this.

[0202] In this embodiment, the second processor includes a low-power space and a non-low-power space. The low-power space and the non-low-power space of the second processor can be understood as storage spaces divided according to power optimization requirements.

[0203] Optionally, the low-power space of the second processor can be a low-power memory space, and the non-low-power space of the second processor can be a non-low-power memory space. The low-power memory space and non-low-power memory space of the second processor can be understood as different memory regions divided according to power optimization requirements. The low-power memory space uses power optimization techniques to reduce power consumption, while the non-low-power memory space focuses more on performance improvement, but may have higher power consumption.

[0204] The relevant information regarding the low-power space, non-low-power space, low-power memory space, and non-low-power memory space of the second processor can be found above and will not be repeated here.

[0205] For example, the low-power space or low-power memory space of the second processor can be selected from low-power storage technologies, such as low-power memory, i.e., the first memory, such as a low-power version of static random access memory (SRAM), or flash memory technology with low-power characteristics. Details regarding low-power memory can be found above and will not be repeated here.

[0206] For example, the non-low-power space or non-low-power memory space of the second processor can be implemented using non-low-power memory, i.e., a second memory, such as DDR or DRAM. Details regarding non-low-power memory can be found above and will not be repeated here.

[0207] In this embodiment, the second processor includes a low-power mode and a non-low-power mode. The low-power mode of the second processor is an operating mode designed to reduce power consumption. In the low-power mode, the second processor may choose to use its low-power memory space (or low-power space) to reduce overall power consumption. In the non-low-power mode, the second processor may choose to use its non-low-power memory space (or non-low-power space), thereby achieving optimal performance and bandwidth.

[0208] Specifically, taking a non-low-power memory space or a non-low-power memory space as an example, when executing the voice interaction method provided in this application embodiment, if the second processor is in low-power mode, it can use the low-power memory space of the second processor, but not the low-power memory space of the second processor. That is, the low-power memory in the low-power memory space is in an active state, while the non-low-power memory in the non-low-power memory space is in a low-power state. When executing the voice interaction method provided in this application embodiment, if the second processor is in non-low-power mode, it can use the non-low-power memory space of the second processor, but not the low-power memory space of the second processor. That is, the low-power memory in the non-low-power memory space is in an active state.

[0209] In this embodiment, the second processor's exit from and entry into low-power mode can be understood as a switching of the operating mode of non-low-power memory (such as DDR) in the non-low-power memory space. Taking non-low-power memory DDR as an example, when the second processor enters low-power mode, the DDR is in a low-power state. When the DDR is in a low-power state, power consumption can be reduced by lowering the operating frequency or entering a low-power mode. When the second processor exits low-power mode, the DDR enters the operating state. When the DDR is in the operating state, it may run at full speed or operate according to the conventional memory access mode to meet the second processor's requirements for high performance and response speed.

[0210] In some embodiments, when the DDR is in a low-power state, the DDR operates with very little current; the DDR may only need to maintain data integrity but cannot read or write data. When the DDR is in a normal state, the DDR operates with a large current. The operating current of the DDR in a low-power state is less than the operating current of the DDR in a normal state.

[0211] In some embodiments, when the DDR is in a low-power state, the DDR operates at a very low voltage, and the DDR may only need to maintain data integrity but cannot read or write data. When the DDR is in a normal state, the DDR operates at a very high voltage. The operating voltage of the DDR in a low-power state is lower than the operating voltage of the DDR in a normal state.

[0212] It is understood that the above example of the first memory being a non-low-power memory DDR can be implemented through the first memory. The embodiments of this application are only illustrated by taking the non-low-power memory DDR as an example, and do not mean that this application is limited to this.

[0213] In this embodiment, by placing the modules that need to run continuously in the preset wake-up module in the low-power memory space, it can be ensured that these modules can still operate normally when the AP is in sleep mode without significantly affecting the overall power consumption. Placing the remaining modules in the preset wake-up module in the non-low-power memory space provides higher performance and bandwidth. High-performance and high-bandwidth memory can process data faster, making the ADSP faster when handling large tasks, running complex programs, or performing multiple tasks. This speed improvement can significantly enhance the efficiency of waking up the smart assistant, thereby improving the user experience.

[0214] The following details the default wake-up module deployed on the ADSP. It is understood that... Figure 4 Taking the deployment of a preset wake-up module on an ADSP as an example does not mean that this application is limited to this. The second processor includes a low-power memory space (or low-power space) and a non-low-power memory space (or non-low-power space), which will Figure 4 The audio digital signal processor in the middle is replaced with a second processor, which can be understood as the deployment of the preset wake-up module in the second processor.

[0215] In this embodiment of the application, the preset wake-up module of the electronic device includes a gesture detection module and a breath wake-up detection module.

[0216] In this embodiment, the gesture detection module is deployed in the low-power memory space of the ADSP, while the breath wake-up detection module is deployed in the non-low-power memory space of the ADSP. The gesture detection module is used to detect the motion state of the electronic device. Specifically, the ADSP runs the gesture detection module to acquire motion data of the electronic device and determines the motion state of the electronic device based on the motion data.

[0217] In this application embodiment, motion data may include, but is not limited to, IMU data. In some embodiments, motion data may include acceleration data in the x-axis direction, acceleration data in the y-axis direction, and acceleration data in the z-axis direction acquired by the accelerometer of an electronic device. In some embodiments, motion data may include angular velocity data.

[0218] Optionally, the gesture detection module may include a pose detection model. Taking the motion data as the acceleration data of an electronic device in the x-axis, y-axis, and z-axis directions, for example, the absolute values ​​of the acceleration data corresponding to the x, y, and z axes (d1, d2, and d3) can characterize whether the electronic device is in motion, p1, p2, and p3 can characterize the amplitude of the electronic device's motion, and the differences between d1 and p1, d2 and p2, and d3 and p3 can characterize the motion state of the electronic device from other dimensions such as the smoothness of motion. Therefore, the pose detection model can comprehensively determine whether the electronic device has been lifted, i.e., whether the electronic device is in a lifted state or a lift event has occurred, based on the above motion data, considering multiple aspects such as whether the electronic device is moving, the amplitude of the movement, and the smoothness of the movement, thereby improving the accuracy of the pose detection model's judgment.

[0219] In this embodiment, the pose detection model can be a trained convolutional neural network model, which may include convolutional layers or fully connected layers.

[0220] It is understood that the pose detection model can also be other neural network models, and this application does not make any specific limitations on this.

[0221] It is understandable that the pose detection model deployed in the low-power memory space of the ADSP can be a lightweight neural network model.

[0222] In some embodiments, the position detection module can be implemented with preset judgment conditions. When the motion data meets the preset judgment conditions, it determines that the electronic device is in a raised state or that a hand-raising event has occurred. The setting of the preset judgment conditions can refer to the relevant content on the triggering timing of the above-mentioned electronic device being in a raised state or having a hand-raising event, and the breath wake-up event. The specific settings can be made according to the actual situation, and this application does not make specific limitations on this.

[0223] In this embodiment of the application, when the user operates the electronic device in a raised state, the user inputs voice into the electronic device. For example... Figure 1F As shown, after the user raises the phone and brings the bottom of the phone close to their mouth, the user inputs the voice command "How's the weather today?" into the phone. That is, the electronic device can receive the user's first voice command "How's the weather today?" when or after the electronic device is in the raised state.

[0224] The breath wake-up detection module is used to perform breath wake-up detection.

[0225] In this embodiment, the gesture detection module detects that the electronic device is in a raised state and notifies the breath wake-up detection module to acquire motion data and / or voice (such as a first voice and / or a second voice). The breath wake-up detection module determines whether to trigger a breath wake-up event based on the acquired motion data and / or voice.

[0226] In some embodiments, if the user's voice is received while the electronic device is in a raised state, the AP is woken up, and the voice interaction module is started or resumed. If the user's voice is not received while the electronic device is not in a raised state, the AP is not woken up, and the voice interaction is neither started nor resumed. In some embodiments, if the user's voice is received while the electronic device is in a raised state, and the third distance between the sound source of the voice and the second microphone is less than the fourth distance between the sound source of the voice and the first microphone, the AP is woken up, but the voice interaction is neither started nor resumed.

[0227] For example, at T1, the gesture detection module detects that the electronic device is in a raised state. The gesture detection module notifies the breath wake-up detection module to acquire motion data and speech (such as a first speech and / or a second speech). At T2, the breath wake-up detection module responds to the notification and acquires motion data and speech. The motion data acquired at T2 can be the motion data generated by the electronic device at T2, and the speech acquired by the breath wake-up detection module at T2 can include the speech signal collected by the microphone of the electronic device at and / or after T2.

[0228] In this embodiment, the gesture detection module can detect whether a breath wake-up event is triggered based on motion data and / or voice. When a breath wake-up event is detected, if the AP is in a sleep state, the AP is woken up, and the voice interaction module is started or resumed through the AP.

[0229] In this embodiment, the breath wake-up detection module may include a preliminary detection model, a voice detection model, a state detection model, and a voice-state detection model. The preliminary detection model determines whether to detect audio data received by the electronic device. The voice detection model determines whether the audio data is a voice command sent by the user to the electronic device. The state detection model determines whether the electronic device is in a raised state when receiving the audio data. The voice-state detection model detects that the audio data received by the electronic device is a voice command and that the electronic device is currently in a raised state.

[0230] It is understandable that the breath wake-up detection module can also wake up the voice interaction module according to other wake-up strategies, and this application does not make any specific limitations on this.

[0231] In the embodiments of this application, the preset wake-up module of the electronic device may further include a breath wake-up activation module.

[0232] The breath wake-up activation module is used to activate the breath wake-up detection module when the electronic device is in the lifted state.

[0233] In this embodiment, the implementation logic of the breath wake-up activation module can be to perform a loop of detection. For example, it can repeatedly check whether the movement state of the electronic device is detected as a raised state until the movement state of the electronic device is detected as a raised state. The breath wake-up activation module can be deployed in the low-power memory space of the ADSP, so as to quickly respond to the movement state of the electronic device being raised.

[0234] In this embodiment, the gesture detection module can be deployed in the sensing area of ​​the low-power memory space. The breath wake-up activation module can be deployed in the audio area of ​​the low-power memory space. The gesture detection module deployed in the sensing area can continue to process sensor data when the AP is in sleep mode, such as determining the motion state of the electronic device by detecting motion data collected by the sensors. The breath wake-up activation module deployed in the audio area can, when the AP is in sleep mode, respond to the electronic device's motion state being raised, activate other modules of the preset wake-up module (such as the breath wake-up detection module) to perform breath wake-up detection.

[0235] In this embodiment, the preset wake-up module of the electronic device may further include a state forwarding module. When the gesture detection module detects that the electronic device is in a raised state, the raised state can be transmitted to the breath wake-up activation module through the state forwarding module.

[0236] In this embodiment, the state forwarding module can be implemented as a message transmission channel between the audio region and the sensing region. This message transmission channel is used to realize inter-process or inter-system communication. For example, when the preset wake-up module is deployed on an ADSP, the state forwarding module can be implemented through a message transmission channel such as qsocket or Qualcomm Messaging Interface (QMI). When the preset wake-up module is deployed on other types of second processors, the state forwarding module can also be implemented through shared memory, etc. It is understood that this application does not specifically limit the specific implementation of the state forwarding module deployed on the second processor.

[0237] In this embodiment, the state forwarding module can be deployed in the non-low-power memory space of the ADSP.

[0238] In this embodiment, the preset wake-up module of the electronic device may further include an audio data storage module. The audio data storage module is used to store the voice collected by the electronic device, so that after the voice interaction module is subsequently woken up, the voice in the audio data storage module can be transmitted to the voice interaction module for voice recognition.

[0239] In this embodiment, the gesture detection module detects that the electronic device is in a raised state and notifies the breath wake-up detection module to acquire motion data and / or voice (such as first voice and / or second voice), and also notifies the audio data storage module to acquire voice (such as first voice and / or second voice). That is, after detecting that the electronic device is in a raised state, it begins to acquire voice signals collected by multiple microphones (such as first microphone and second microphone), and copies the acquired voice signals into two copies, one of which is transmitted to the breath wake-up detection module and the other to the audio data storage module.

[0240] For example, the audio data access module can be implemented as audio data data access management (DAM).

[0241] In this embodiment, the audio data access module can be deployed in the non-low-power memory space of the ADSP.

[0242] In some embodiments, the audio data access module can be a module independent of the preset wake-up module.

[0243] In this embodiment, the preset wake-up module of the electronic device may further include a breath wake-up pre-module. The breath wake-up pre-module is responsible for the pre-processing of breath wake-up detection. When the breath wake-up activation module detects that the electronic device is in a lifted state, the breath wake-up activation module activates the breath wake-up pre-module.

[0244] In this embodiment, when the breath wake-up activation module detects that the electronic device is in a lifted state, the breath wake-up activation module activates the breath wake-up pre-module. The breath wake-up pre-module then controls the ADSP to exit the low-power mode, acquires the motion data collected by the sensor, notifies the audio data storage module to cache the audio data, and activates the breath wake-up detection module.

[0245] In this embodiment, the breath wake-up pre-module may include a low-power exit unit, a motion data acquisition unit, a notification unit, and a startup unit. The low-power exit unit controls the ADSP to exit low-power mode. The motion data acquisition unit acquires motion data collected by the sensor. The notification unit notifies the audio data storage module to perform audio data caching. The notification unit can notify the audio data storage module to perform data caching by modifying its flag. The startup unit starts the breath wake-up detection module.

[0246] In this embodiment, the breath wake-up pre-module can be deployed in both the low-power memory space and the non-low-power memory space of the ADSP. For example, the exit low-power unit is deployed in the low-power memory space of the ADSP. Alternatively, the exit low-power unit can be deployed in the audio region of the low-power memory space. Furthermore, the motion data acquisition unit, notification unit, and startup unit can be deployed in the non-low-power memory space of the ADSP.

[0247] In this embodiment, the preset wake-up module may further include a detection processing module (not shown). The detection processing module is used to perform corresponding operations based on the detection results of the breath wake-up detection module. When the breath wake-up detection module detects successful breath wake-up, that is, determines that a breath wake-up event has been triggered, it executes the operation corresponding to the successful detection, such as waking up the AP and clearing the data and flag bits used for the current breath wake-up detection. When the breath wake-up detection module detects failed breath wake-up, that is, determines that a breath wake-up event has not been triggered, it executes the operation corresponding to the failed detection, such as clearing the data and flag bits used for the current breath wake-up detection and controlling the ADSP to enter a low-power mode.

[0248] For example, when the breath wake-up detection module detects a breath wake-up event, it wakes up the AP and modifies the flag of the audio data access module to control the audio data access module to close the audio data cache.

[0249] As another example, when the breath wake-up detection module detects that the breath wake-up event has not been triggered, it controls the ADSP to enter a low-power mode and modifies the flag bit of the audio data access module to control the audio data access module to turn off the audio data buffer.

[0250] In some embodiments, the detection processing module can control the ADSP to enter a low-power mode before performing breath wake-up detection.

[0251] In some embodiments, the detection processing module may control the audio data access module to disable the audio data cache before performing breath wake-up detection.

[0252] The detection and processing module can operate independently of the breath wake-up detection module, or it can be integrated into the breath wake-up detection module or as part of other modules. The detection and processing module can be deployed in the non-low-power memory space of the ADSP.

[0253] In this embodiment, the gesture detection module detects that the electronic device is in a raised state. Through the aforementioned state forwarding module, breath wake-up initiation module, and breath wake-up pre-processor module, it notifies the breath wake-up detection module to acquire motion data and voice (such as a first voice and / or a second voice), and notifies the audio data access module to acquire voice (such as a first voice and / or a second voice). In this embodiment, the various modules of the preset wake-up module are distributed and deployed in the ADSP. Specifically, the gesture detection module and the breath wake-up initiation module are deployed in the low-power memory space of the ADSP, while the state forwarding module, the breath wake-up pre-processor module, the breath wake-up detection module, and the audio data access module are deployed in the non-low-power memory space of the ADSP, reducing the preset wake-up module's dependence on the ADSP's low-power memory space.

[0254] The gesture detection module and the breath wake-up module implement complex programs that are not complicated and occupy little low-power memory space. Even if the low-power memory space of the ADSP is limited, it is still possible to deploy the gesture detection module and the breath wake-up module in the low-power memory space of the ADSP.

[0255] By deploying the gesture detection module and the breath wake-up activation module in the low-power memory space of the ADSP, the gesture detection module and the breath wake-up activation module can detect the motion status of the electronic device in real time while the AP is in sleep mode, ensuring real-time processing of the wake-up voice interaction module and meeting the high real-time requirements of wake-up.

[0256] Deploying the breath wake-up detection module in the non-low-power memory space of the ADSP, where the space is large enough to accommodate the module, and leveraging the high performance and bandwidth provided by the ADSP's non-low-power memory space, can improve the detection efficiency of the breath wake-up detection module.

[0257] The low-power exit unit of the breath wake-up pre-module is deployed in the low-power memory space of the ADSP, so that the ADSP can be quickly controlled to exit the low-power mode when the breath wake-up start-up module detects the lift state.

[0258] The state forwarding module is deployed in the non-low-power memory space of the ADSP to quickly forward the hand-raise state based on the high performance and bandwidth provided by the non-low-power memory space of the ADSP.

[0259] The motion data acquisition unit, notification unit, and start-up unit of the breath wake-up pre-module are deployed in the non-low-power memory space of the ADSP, so as to quickly trigger the breath wake-up detection process based on the high performance and bandwidth provided by the non-low-power memory space of the ADSP.

[0260] The audio data access module is deployed in the ADSP’s non-low-power memory space to provide data caching efficiency based on the high performance and bandwidth offered by the ADSP’s non-low-power memory space.

[0261] Taking SRAM as the low-power memory in the low-power memory space and DRAM as the non-low-power memory in the non-low-power memory space as an example, and using IMU data acquired by the gesture detection module as the motion data collected by the IMU, please refer to [link to relevant documentation]. Figure 5 The working process of the preset wake-up module provided in the embodiments of this application is illustrated by way of example.

[0262] It is understood that, taking the first memory as a non-low-power memory DRAM as an example, all the functions of the non-low-power memory DRAM can be implemented through the first memory. Similarly, taking the second memory as a low-power memory SRAM as an example, all the functions of the low-power memory SRAM can be implemented through the second memory. The embodiments of this application are only illustrated with non-low-power memory DRAM and low-power memory SRAM as examples, and do not mean that this application is limited to this.

[0263] The IMU of the electronic device acquires IMU data in real time.

[0264] The gesture detection module acquires IMU data collected by the IMU. When the gesture detection module detects a hand-raise event based on the IMU data, it sends the hand-raise event to the state forwarding module.

[0265] The state forwarding module determines that the electronic device is in the raised state in response to a hand-raise event. Specifically, the state forwarding module can obtain the hand-raise event through inter-process or inter-system communication, and then determine that the electronic device is in the raised state in response to the hand-raise event. The state forwarding module sends the raised state to the breath wake-up initiation module.

[0266] The breath wake-up activation module responds to the electronic device being lifted, triggering the breath wake-up pre-module to start. Upon startup, the breath wake-up pre-module performs the following operations: 1. Controls the ADSP to exit low-power mode; 2. Acquires IMU data and transmits it to the breath wake-up detection module; 3. Notifies the audio data storage module to buffer audio data; 4. Starts the breath wake-up detection module to perform breath wake-up detection.

[0267] After the breath wake-up detection module is activated, it acquires audio data collected by the first and second microphones. The breath wake-up detection module also acquires IMU data transmitted from the breath wake-up pre-amplifier module.

[0268] After the audio data access module is activated, it acquires and stores the audio data collected by the first and second microphones. This stored audio data is the first audio data, which is the voice data of the first speech. In response to the electronic device being in a raised state, the audio data collected by the first and second microphones is transmitted to the breath wake-up detection module and then to the audio data access module. The audio data stored in the audio data access module is the first audio data.

[0269] When the breath wake-up detection module detects a breath wake-up event based on the acquired audio data and / or motion data (such as IMU data), it wakes up the application processor (AP). The smart assistant is then launched or woken up via the AP.

[0270] In some embodiments, after the breath wake-up detection module detects a breath wake-up event, the breath wake-up detection module is also used to control the audio data access module to close the data cache.

[0271] After the smart assistant is activated or woken up, it can retrieve the first audio data cached by the audio data storage module and perform speech recognition and corresponding operations based on the first audio data. For example, in Figure 1G In this system, after waking up the smart assistant, the electronic device's display screen activates and shows a first interface. This first interface includes content related to "What's the weather like today?" based on the first audio data. The smart assistant can also perform corresponding operations based on voice commands derived from the first audio data, such as... Figure 1H As shown, the electronic device responds to the voice command "How's the weather today?" and displays the corresponding weather forecast information on the screen.

[0272] For example, the process by which the smart assistant processes the first audio data cached by the audio data access module may include: after waking up the application processor (AP), the hardware abstraction layer of the application processor (AP) transmits the first audio data cached by the audio data access module to the framework layer. The audio recording module of the framework layer reads the first audio data. The audio recording module of the framework layer transmits the first audio data to the smart assistant, and the smart assistant performs speech recognition on the first audio data.

[0273] In some embodiments, the smart assistant creates an instance of the AudioRecord class, configuring the required audio source, sample rate, channel configuration, audio format, and buffer size. The smart assistant calls the `startRecording()` method of the AudioRecord instance to begin recording. The AudioRecord object captures audio data from the microphone at the frame layer and stores it in an internal buffer. At the application layer, the smart assistant reads audio data from the AudioRecord's buffer by calling the `read()` method and performs speech recognition on the audio data.

[0274] During the process of the intelligent assistant performing speech recognition on the received audio data, the intelligent assistant generates feedback information, which may be text, speech, or other forms, and transmits it to the user through the framework layer and hardware abstraction layer. For example... Figure 1G and Figure 1H The corresponding text is displayed on the screen.

[0275] When deploying the various modules of the preset wake-up module in a specific storage space or region, the preset wake-up module can be divided into sub-modules, each with a clearly defined function and responsibility. As mentioned above, the sub-modules of the preset wake-up module may include a gesture detection module, a breath wake-up pre-module, a breath wake-up initiation module, a status forwarding module, a breath wake-up detection module, and an audio data access module. Specific storage space is allocated to each sub-module. For example, a low-power sensing area is allocated to the gesture detection module, non-low-power space is allocated to the status forwarding module, the breath wake-up detection module, and the audio data access module, a low-power audio area is allocated to the breath wake-up initiation module, and an audio area and non-low-power space are allocated to the breath wake-up pre-module. Code is written to implement the function of each sub-module. The code is then deployed to the space allocated to each sub-module, such as memory space.

[0276] The following describes the specific implementation of each module in the preset wake-up module.

[0277] In this embodiment of the application, the gesture detection module is deployed in the sensing area of ​​the low-power space, which can be implemented such that the code for implementing the functions related to the gesture detection module runs in the sensing area of ​​the low-power space.

[0278] In this embodiment, the state forwarding module may specifically include: 1. Registering the client with the sensor server and registering the hand-raise event. 2. When the gesture detection module detects the hand-raise event, it exits the low-power mode and sends the hand-raise event to the sensor server. The sensor server notifies the client of the hand-raise event. After receiving the hand-raise event, the client notifies the breath wake-up module that the current electronic device is in a raised state by modifying a global variable.

[0279] For example, the implementation of the state forwarding module may include: initializing client instances related to the hand-raise event through a function, wherein the function carries corresponding parameters, such as the client that obtains the hand-raise event and a callback function. Specifically, the client that obtains the hand-raise event can be indicated by `freewake_req_motion_client_handle`, and the hand-raise event or the detection of a raised state can be obtained through the `Freewake_req_client_int_cb` callback function.

[0280] Specifically, when the gesture detection module detects a raise event, it exits low-power mode. The gesture detection module sends the raise event to the sensor server, which then notifies the audio client of the raise event via the callback function `Freewake_req_client_int_cb`. The callback function `Freewake_req_client_int_cb` runs in the audio client. The audio client parses the event information using this callback function, and if the raise event is confirmed, it sets the global variable `motion-state` to "true" (true, non-zero, non-empty, etc.). `motion-state` indicates the motion state of the electronic device. When `motion-state` is set to "true", it indicates that the electronic device is in a raised state. When `motion-state` is set to "false", it indicates that the electronic device is not in a raised state.

[0281] For example, deploying the state forwarding module in a non-low-power space can be implemented such that the code used to implement the functions related to the state forwarding module runs in the non-low-power space.

[0282] In this embodiment of the application, the breath wake-up module can be implemented to cyclically acquire the motion state of the electronic device until the operating state of the electronic device is obtained as the lifted state.

[0283] For example, the code implementation of the breath wake-up startup module may include:

[0284] If (!get_motion_status())

[0285] The "motion_status" in the `get_motion_status()` function is received by `freewake_req_motion_client_handle` during the registration of the hand-raise event. Upon detecting a hand-raise event, the status forwarding module sets `motion_status = true`. The `if(!get_motion_status)` method checks if the return value of a variable or function call named `get_motion_status` is "false" (e.g., false, 0, null, undefined). The `if(!get_motion_status)` method also checks if no hand-raise event was detected; if not, it loops through `if(!get_motion_status)`. Finally, if no hand-raise event was detected, the breath-wake pre-module is triggered.

[0286] For example, the breath wake-up activation module deployed in the audio region of the low-power space can be implemented as code for implementing the functions related to the breath wake-up activation module, including but not limited to the code of If(!get_motion_status()) mentioned above, running in the audio region.

[0287] In this embodiment, the breath wake-up pre-module includes the following operations: exiting low-power mode, acquiring motion data, notifying the audio data storage module to cache data, and notifying the breath wake-up detection module to perform breath wake-up detection.

[0288] For example, the specific implementation of exiting low-power mode (exiting low-power unit) may include: calling a function to notify the ADSP to exit low-power mode when a lift-off state is detected. For example, calling a function to notify the aforementioned low-power island to exit low-power mode.

[0289] For example, acquiring motion data can be done by first initializing the data and then retrieving it through a function. Taking ACC data as an example, the code implementation for acquiring ACC data (motion data acquisition unit) can include:

[0290] capi_client_accel_init();

[0291] get_accel_data(accel_data)

[0292] In other words, it can be initialized using the function capi_client_accel_init(), and then ACC data can be obtained using the function get_accel_data(accel_data).

[0293] For example, notifying the audio data access module to perform data caching can be achieved by modifying a flag. After detecting the corresponding flag, the audio data access module performs data caching. For example, the code implementation for notifying the audio data access module to perform data caching (notification unit) may include:

[0294] flow_ctrl_cfg.is_need_mock_read=FALSE;

[0295] capi_freewakeup_send_ctrl_msg_flow_to_dam(me_ptr,&flow_ctrl_cfg)

[0296] The `flow_ctrl_cfg.is_need_mock_read` flag is used by the audio data access module. This flag is modified by assigning it a second value, such as `FALSE`. The modified flag is then passed to the audio data access module via the `capi_freewakeup_send_ctrl_msg_flow_to_dam` function. When the audio data access module detects that the flag is `FALSE`, it performs audio data buffering, meaning it acquires the data collected by the microphone and caches it in the buffer. When the audio data access module detects that the flag is `TRUE`, it does not perform audio data buffering.

[0297] For example, the breath wake-up pre-module can be deployed in a non-low-power space and an audio region, such that the code for exiting low-power mode runs in the audio region, and the code for acquiring motion data and notifying the audio data access module to cache data runs in the non-low-power space.

[0298] For example, the implementation of the breath wake-up initiation module and the breath wake-up pre-module may include:

[0299] If(!get_motion_status()){return CAPI_EOK;

[0300] }

[0301] {Notify the user to exit low-power mode;}

[0302] capi_client_accel_init();

[0303] get_accel_data(accel_data);

[0304] flow_ctrl_cfg.is_need_mock_read=FALSE;

[0305] capi_freewakeup_send_ctrl_msg_flow_to_dam(me_ptr,&flow_ctrl_cfg)}

[0306] When get_motion_status() returns "false", it returns CAPI_EOK; when get_motion_status() returns "true", indicating that the electronic device is in the raised state, the following code is executed (that is, the content of the breath wake-up pre-module):

[0307] {Notify the user to exit low-power mode;}

[0308] capi_client_accel_init();

[0309] get_accel_data(accel_data);

[0310] flow_ctrl_cfg.is_need_mock_read=FALSE;

[0311] capi_freewakeup_send_ctrl_msg_flow_to_dam(me_ptr,&flow_ctrl_cfg)}.

[0312] In this embodiment of the application, when the breath wake-up detection module successfully recognizes the breath wake-up event, the electronic device (detection and processing module) can perform the following operations: 1. Clear the lift-up state; 2. Clear the accelerometer (accelerometer, i.e., acceleration sensor) service related to the client; 3. Wake up the AP and voice interaction module; 4. Notify the audio data access module to close the data cache.

[0313] For example, the function Clear_motion_status() can be used to clear the operating status of an electronic device from the lifted state.

[0314] For example, the client-related accelerometer (accel, i.e., acceleration sensor) service can be initialized or cleaned up using the function capi_client_accel_deinit().

[0315] For example, notifying the audio data access module to disable the data cache can be achieved by modifying a flag. Upon detecting the corresponding flag, the audio data access module disables the data cache. For example, the code implementation for notifying the audio data access module to disable the data cache may include:

[0316] capi_freewakeup_set_param;

[0317] flow_ctrl_cfg.is_need_mock_read=TRUE;

[0318] capi_freewakeup_send_ctrl_msg_flow_to_dam(me_ptr,&flow_ctrl_cfg).

[0319] For example, when the breath wake-up detection module determines that the voice interaction module needs to be woken up, the ADSP wakes up the AP, and the AP can start or resume the smart assistant. For instance, when it is necessary to wake up the smart assistant, the AP can pass wake-up information to the smart assistant through the HAL layer to activate the smart assistant.

[0320] In this embodiment of the application, if the breath wake-up detection module fails to detect the breath, the electronic device (detection processing module) can perform the following operations: 1. Notify the audio data access module to close the data cache; 2. Clear the lift-up state; 3. Clear the accelerometer (accelerometer sensor) service related to the client; 4. Enter the low power mode.

[0321] The code implementation for notifying the audio data access module to close the data cache, clear the lift-up state, and clean up the accelerometer service related to the client can refer to the above.

[0322] For example, entering a low-power mode can be specifically implemented by calling a function to notify the ADSP to enter a low-power mode. For instance, calling a function to notify the aforementioned low-power island to enter a low-power mode.

[0323] It is understood that the specific implementations of the above modules are merely examples, and this application does not impose any specific limitations on them.

[0324] The following exemplary description illustrates the power consumption of an electronic device when executing the voice interaction method provided in the embodiments of this application (i.e., the operation of the preset wake-up module provided in the embodiments of this application).

[0325] like Figure 6AAs shown, during 0-T1, the screen of the electronic device is off, the electronic device is not lifted, the gesture detection module and the breath wake-up module deployed in the low-power memory space of the ADSP are both started, the ADSP runs the gesture detection module and the breath wake-up module deployed in the low-power memory space, and the power consumption of the electronic device is the first value.

[0326] In some embodiments, during 0-T1, the screen of the electronic device is off, the electronic device is not lifted, and the power consumption of the electronic device is within a first power consumption range.

[0327] At time T1, the screen of the electronic device is off. The electronic device is lifted up, the gesture detection module detects the hand lift event, and the power consumption of the electronic device changes from the first value to the second value.

[0328] During T1-T2, based on the gesture detection module detecting a hand-raise event, the ADSP exits low-power mode and transmits the hand-raise event to the state forwarding module. The ADSP runs the state forwarding module deployed in the non-low-power memory space. In response to the hand-raise event, the state forwarding module obtains the motion state of the electronic device as the raised state, and the power consumption of the electronic device increases from the first value to the second value.

[0329] In some embodiments, during T1-T2, the screen of the electronic device is off, the electronic device is lifted, and the power consumption of the electronic device is within the second power consumption range.

[0330] At time T2, the status forwarding module sends the hand-raise status to the breath wake-up activation module, and the ADSP enters low-power mode. From T2 to T3, the ADSP is in low-power mode, and the ADSP runs the breath wake-up activation module deployed in the low-power memory space, with the power consumption of the electronic device at the first value.

[0331] In some embodiments, the screen of the electronic device is off, the electronic device is lifted, and during the brief period from T2 to T3, the power consumption of the electronic device is within a first power consumption range.

[0332] At time T3, the breath wake-up activation module achieves a raised state. After time T3, based on achieving the raised state, the breath wake-up activation module activates the breath wake-up pre-module. The ADSP runs the exit low-power unit deployed in the low-power memory space, triggering the ADSP to exit low-power mode. The ADSP then runs the motion data acquisition unit, notification unit, and activation unit deployed in the non-low-power memory space, and the power consumption of the electronic device is the second value.

[0333] In some embodiments, at time T3, the screen of the electronic device is off, the electronic device is lifted, and the power consumption of the electronic device is within the second power consumption range.

[0334] After time T3, the breath wake-up pre-module starts the audio data access module and the breath wake-up detection module. The ADSP runs the audio data access module and the breath wake-up detection module deployed in the non-low-power memory space, and the power consumption of the electronic device is the second value.

[0335] In some embodiments, after time T3, the screen of the electronic device is off, the electronic device is lifted, and the power consumption of the electronic device is within the second power consumption range.

[0336] In this embodiment of the application, after time T3, if the breath wake-up detection module determines that a breath wake-up event has been triggered, wakes up the AP, and starts or resumes the voice interaction module, then when the AP is woken up, the electronic device screen lights up, and the power consumption of the electronic device is the third value.

[0337] In some embodiments, after time T3, the AP is woken up, the electronic device screen lights up, and the power consumption of the electronic device is within the third power consumption range.

[0338] In this embodiment of the application, after time T3, if the breath wake-up detection module determines that the breath wake-up event has not been triggered, the screen of the electronic device is in an off state, and the power consumption of the electronic device is the first value.

[0339] In some embodiments, after time T3, if the AP is not woken up, the screen of the electronic device is off, and the power consumption of the electronic device is within the first power consumption range.

[0340] In this embodiment, the value of the first power consumption range is less than the value of the second power consumption range. The value of the second power consumption range is less than the value of the third power consumption range. Exemplarily, the first power consumption range may include one or more first power consumptions, the second power consumption range may include one or more second power consumptions, and the third power consumption range may include one or more third power consumptions, wherein any first power consumption is less than any second power consumption, and any second power consumption is less than any third power consumption.

[0341] In this embodiment, when the power consumption of the electronic device is a first value or within a first power consumption range, and the voltage of the electronic device is 4.2V, the current of the electronic device can be 5-15mA. When the power consumption of the electronic device is a second value or within a second power consumption range, and the voltage of the electronic device is 4.2V, the current of the electronic device can be 25-35mA. When the power consumption of the electronic device is a third value or within a third power consumption range, and the voltage of the electronic device is 4.2V, the current of the electronic device is in the hundreds or thousands, such as hundreds of mA or thousands of mA. It is understood that when the value in the first power consumption range is less than the value in the second power consumption range, and the value in the second power consumption range is less than the value in the third power consumption range, the above power consumption ranges are for reference only, and the actual power consumption of the electronic device may vary depending on the device model, brand, usage environment, and specific application scenario.

[0342] It is understandable that in practical applications, the breath-activated wake-up module may have a short running time, such as a short time interval between T2 and T3, or it may be limited by the accuracy of the power consumption data acquisition of the electronic device. For example, the power consumption data acquired by the electronic device may be... Figure 6B or Figure 6C As shown.

[0343] like Figure 6B As shown, during time 0-T11, both the gesture detection module and the breath wake-up activation module, deployed in the low-power memory space of the ADSP, are started. Both modules operate within the low-power memory space, and the power consumption of the electronic device is within the first power consumption range. At time T11, the gesture detection module detects a hand-raising event.

[0344] like Figure 6B As shown, during 0-T11, the screen of the electronic device is off, the electronic device is not lifted, the gesture detection module and the breath wake-up module deployed in the low-power memory space of the ADSP are both started, the ADSP runs the gesture detection module and the breath wake-up module deployed in the low-power memory space, and the power consumption of the electronic device is the first value.

[0345] In some embodiments, during 0-T11, the screen of the electronic device is off, the electronic device is not lifted, and the power consumption of the electronic device is within a first power consumption range.

[0346] At time T11, the screen of the electronic device is off. The electronic device is lifted up, the gesture detection module detects the hand lift event, and the power consumption of the electronic device changes from the first value to the second value.

[0347] In some embodiments, at time T11, the screen of the electronic device is off, the electronic device is lifted, and the power consumption of the electronic device is within the second power consumption range.

[0348] In T11-T12, based on the gesture detection module detecting a hand-raise event, the ADSP exits low-power mode and transmits the hand-raise event to the state forwarding module. The ADSP runs the state forwarding module deployed in the non-low-power memory space. Responding to the hand-raise event, the state forwarding module obtains the electronic device's motion state as "raised," and sends this state to the breath wake-up activation module. The electronic device's power consumption is the second value or within the second power consumption range. The ADSP then runs the breath wake-up activation module deployed in the low-power memory space. Due to the short running time of the breath wake-up activation module, or limitations in the electronic device's power consumption acquisition accuracy, the acquired power consumption of the electronic device remains the second value or still within the second power consumption range.

[0349] The breath wake-up activation module activates the breath wake-up pre-module based on the obtained lift-up state. The ADSP runs the exit low-power unit deployed in the low-power memory space, triggering the ADSP to exit low-power mode. The ADSP runs the motion data acquisition unit, notification unit, and activation unit deployed in the non-low-power memory space. The ADSP runs the audio data access module and breath wake-up detection module deployed in the non-low-power memory space. The power consumption of the electronic device is the second value or within the second power consumption range.

[0350] Between T11 and T12, electronic devices can receive the user's voice.

[0351] At time T12, the breath wake-up detection module determines that no breath wake-up event has been triggered. Since the ADSP does not wake up the AP, the AP is in sleep mode, the ADSP enters low-power mode, and the power consumption of the electronic device is the first value.

[0352] In some embodiments, the breath wake-up detection module determines that the absence of a breath wake-up event is within a first power consumption range.

[0353] Figure 6B and Figure 6C The differences include, Figure 6C During time T12, the breath wake-up detection module determines that a breath wake-up event has been triggered, waking up the AP. The AP then exits sleep mode, the electronic device's screen illuminates, and the AP starts or resumes the voice interaction module. The voice interaction module can process the voice received between T11 and T12, and the electronic device displays the corresponding interface (e.g., ...). Figure 1G or Figure 1H As shown in the figure, the power consumption of the electronic device is the third value.

[0354] In some embodiments, after T12, the electronic device lights up its screen, and the power consumption of the electronic device is in the third power consumption range.

[0355] Figure 6B and Figure 6AThe differences include that after the gesture detection module detects the lift-up state, Figure 6B The power consumption of the electronic device shown is either the second power consumption or within the second power consumption range, until it is determined that no breath wake-up event has been triggered. When it is determined that no breath wake-up event has been triggered, the power consumption of the electronic device decreases from the second value to the first value, or the power consumption of the electronic device decreases from the second power consumption range to the first power consumption range.

[0356] Figure 6C and Figure 6A The differences include that after the gesture detection module detects the lift-up state, Figure 6B The power consumption of the electronic device shown is either the second power consumption or within the second power consumption range, until a breath wake-up event is determined to be triggered. When a breath wake-up event is determined to be triggered, the power consumption of the electronic device decreases from the second value to the first value, or the power consumption of the electronic device increases from the second power consumption range to the third power consumption range.

[0357] It is understandable that the actual power consumption of the electronic device is not constant, but fluctuates within a reasonable range. For example, in the first detection: the screen of the electronic device is off and the device is not lifted, the power consumption is Pa1. When the screen is off and the device is lifted, the power consumption is Pb1. After the device is lifted and receives voice, the screen lights up, and the power consumption is Pc1. In the second detection: the screen is off and the device is not lifted, the power consumption is Pa2. When the screen is off and the device is lifted, the power consumption is Pb2. After the device is lifted and receives voice, the screen lights up, and the power consumption is Pc2. Even if Pa1 is different from Pa2, Pb1 is different from Pb2, and Pc1 is different from Pc2, but Pa1 is less than Pb1 and Pb1 is less than Pc1, and / or Pa2 is less than Pb2 and Pb2 is less than Pc2, it can be determined that the electronic device is executing the voice interaction method provided in this application embodiment (i.e., the preset wake-up module provided in this application embodiment is running).

[0358] The following uses the aforementioned software module as an example to exemplify the timing interaction of the voice interaction method provided in the embodiments of this application.

[0359] Voice interaction methods can be divided into three distinct phases: the initiation phase, the speech recognition phase, and the dialogue phase.

[0360] The startup phase can be understood as activating the preset wake-up module used to wake up the voice interaction module, so that the preset wake-up module in the ADSP is in a continuous running state. When a breath wake-up event is detected, the AP is woken up, and the smart assistant is started or restored.

[0361] When the aforementioned preset wake-up module is in a continuous running state, some modules of the preset wake-up module can run. For example, some modules of the preset wake-up module deployed in the low-power memory space of the ADSP can run, while other modules of the preset wake-up module deployed in the non-low-power memory space of the ADSP cannot run. Alternatively, some modules of the preset wake-up module deployed in the low-power memory space of the ADSP cannot run, while other modules of the preset wake-up module deployed in the non-low-power memory space of the ADSP can run.

[0362] The speech recognition stage can be understood as the stage where the smart assistant performs speech recognition and determines whether the preset wake-up module should start or stop operating based on the speech recognition status. The dialogue stage can be understood as the process of recording a conversation between the user and the smart assistant. For example, after successful voiceprint recognition, the conversation is recorded.

[0363] Start-up phase

[0364] like Figure 7 As shown, the startup phase of the voice interaction method provided in this application embodiment is illustrated, and it is applied to an electronic device. The startup phase of the voice interaction method may include, but is not limited to, the following steps:

[0365] S11. Electronic devices activate the breath wake-up function.

[0366] In this embodiment, the electronic device activates a breath-activated wake-up function. The electronic device can activate the breath-activated wake-up function in response to a user's operation. For example... Figure 1E As shown, the electronic device activates the breath wake-up function in response to the user's input operation on the switch control 105. In some embodiments, the electronic device has the breath wake-up function enabled by default. For example, if the switch control 105 is enabled by default, the breath wake-up function is activated when the electronic device is powered on.

[0367] It is understandable that when an electronic device activates the breath-activated wake-up function, it can wake up the voice interaction module through a preset wake-up module. For example, the preset wake-up module running on the ADSP detects that a breath-activated wake-up event has been triggered, and the ADSP wakes up the AP. The AP then wakes up the voice interaction module, such as to start or resume the voice interaction module. For details on breath-activated wake-up events and waking up the voice interaction module, please refer to the above.

[0368] For example, taking the Android system as an example, S11 may include S111-S115.

[0369] S111, the smart assistant activates the preset wake-up module.

[0370] Taking the voice interaction module as an example of a smart assistant, when an electronic device activates the breath wake-up function, the smart assistant activates the preset wake-up module.

[0371] S112, the smart assistant sends a request to the framework layer to activate the preset wake-up module.

[0372] The smart assistant initiates the preset wake-up module by sending a request to the framework layer to initiate the preset wake-up module, such as sending a request to the audio trigger module of the framework layer to initiate the preset wake-up module.

[0373] S113, the framework layer sends a request to the audio digital signal processor to load the model.

[0374] The request to load a model is used to instruct the loading of relevant models and / or modules of a preset wake-up module. These relevant models and / or modules may include, but are not limited to: position detection modules, breath wake-up initiation modules, pose detection models, preliminary detection models, speech detection models, state detection models, and speech-state detection models.

[0375] The framework layer can respond to a request to activate a preset wake-up module by sending a model loading request to the hardware abstraction layer, which in turn sends the model loading request to the audio digital signal processor (ADSP). For example, the audio triggering module in the framework layer sends a model loading request to the audio triggering module in the hardware abstraction layer, which then sends the model loading request to the ADSP.

[0376] S114, Audio Digital Signal Processor Loading Model.

[0377] The Audio Digital Signal Processor (ADSP) responds to a model loading request by loading a model, such as loading relevant models and / or modules. Loading relevant models and / or modules by the ADSP may include, but is not limited to: requesting storage space from relevant models and / or modules, loading relevant models and / or modules, and initializing the microphone, etc. Requesting storage space from relevant models and / or modules may include: requesting non-low-power memory space in the ADSP from the audio data access module, and requesting low-power memory space in the ADSP from the gesture detection module and the breath wake-up activation module, etc.

[0378] For example, ADSP can load a model in response to a request to load a model via loadSoundModel() or loadPhraseSoundModel().

[0379] In this embodiment, when the ADSP loads the relevant model and / or module in response to the request to load the model, it registers the client with the sensor server and registers the hand-raise event.

[0380] For example, the state forwarding module running on ADSP registers clients with the sensor server and registers raise-to-hand events. Client instances related to the raise-to-hand event can be initialized via a function that carries relevant parameters, such as the client to retrieve the raise-to-hand event and a callback function. Specifically, the client to retrieve the raise-to-hand event can be indicated by `freewake_req_motion_client_handle`, and the raise-to-hand event can be retrieved via the `Freewake_req_client_int_cb` callback function.

[0381] S115, the framework layer sends a request to the audio digital signal processor to initiate breath wake-up recognition.

[0382] In this embodiment, the framework layer sends a request to the ADSP to initiate breath wake-up recognition. For example, the audio triggering module of the framework layer sends a request to the hardware abstraction layer to initiate breath wake-up recognition, and the hardware abstraction layer then sends the request to the ADSP to initiate breath wake-up recognition.

[0383] For example, the request to initiate breath wake-up recognition can be sent through the function startRecognition. StartRecognition allows the ADSP to run the gesture detection module and the breath wake-up initiation module. startRecognition carries a callback function HandlesessionCallBack. It should be noted that steps S111-S115 above are only illustrative of the internal implementation using the Android architecture as an example, and do not imply that the embodiments of this application are limited to this. Optionally, steps S111-S115 above can be implemented by one or more internal modules of a system. For example, they can all be implemented by an application at the application layer, or by the system's main process, or by a low-level driver, such as a Windows system or an iOS system, etc. This application does not impose any limitations on this. Step S12: The electronic device detects breath wake-up.

[0384] In this embodiment of the application, after the electronic device activates the breath wake-up function, the electronic device detects breath wake-up to wake up the AP and start or restore the smart assistant when a breath wake-up event is detected.

[0385] For example, step S12 may include steps S116 to S117. In some embodiments, step S11 may include steps S111 to S113, and step S12 may include steps S114 to S117.

[0386] S116, Audio Digital Signal Processor detects breath wake-up events.

[0387] In this embodiment, the audio digital signal processor (ADSP) detects whether a breath wake-up event is triggered. Specifically, the preset wake-up module of the ADSP performs breath wake-up recognition and detects whether a breath wake-up event is triggered. The details of the preset wake-up module's breath wake-up recognition and detection of whether a breath wake-up event is triggered can be found above. The relevant process of the ADSP performing breath wake-up recognition will be described in detail below.

[0388] S117, When a breath wake-up event is detected, the audio digital signal processor reports the breath wake-up event to the frame layer.

[0389] In this embodiment of the application, when the ADSP's breath wake-up detection module performs breath wake-up recognition and detects a breath wake-up event, the ADSP reports the breath wake-up event to the framework layer.

[0390] Among them, the breath wake-up event can be used to wake up the AP, and after the AP is woken up, the smart assistant can be started or restored through the AP.

[0391] In this embodiment, the Hardware Abstraction Layer (HAL) can receive breath wake-up events from the ADSP via the callback function HandlesessionCallBack. The HAL then reports the breath wake-up events to the audio triggering module of the framework layer.

[0392] It should be noted that the above steps S116-S117 are only used as an example of the Android architecture to illustrate the internal implementation, and do not mean that the embodiments of this application are limited to this. Optionally, the above steps S116-S117 can be implemented by one or more internal modules / devices of a system, for example, all of which are executed by a second processor.

[0393] Step S13: When a breath wake-up event is detected, the electronic device performs voice recognition.

[0394] In this embodiment, when the electronic device detects a breath wake-up event, it wakes up the AP and starts or resumes the smart assistant, then uses the smart assistant for voice recognition.

[0395] For example, step S13 may include steps S118 to S119.

[0396] S118, the framework layer initiates speech recognition.

[0397] Optionally, in conjunction with steps S116-117, when the framework layer receives a breath wake-up event, the breath wake-up event reported by the framework layer triggers the smart assistant to start voice recognition.

[0398] In this embodiment, upon receiving a breath wake-up event, the audio triggering module of the framework layer can determine whether the smart assistant needs to be started or resumed. Therefore, the audio triggering module of the framework layer can control the smart assistant to start or resume. In response to the wake-up event, the audio triggering module wakes up the AP, starts or resumes the smart assistant through the AP, and sends a message to the smart assistant to initiate speech recognition, thereby launching the speech recognition process.

[0399] S119, the smart assistant performs voice recognition.

[0400] In this embodiment of the application, after the smart assistant is started or resumed, the smart assistant obtains the voice (such as the first voice) cached by the audio data access module and performs voice recognition on the voice.

[0401] In this embodiment of the application, after the smart assistant is activated, the electronic device can display a user interface, such as... Figure 1G and Figure 1H As shown, when the intelligent assistant performs voice recognition, it can display a corresponding interface related to the first voice. This application does not limit the specific implementation method of the user interface.

[0402] The user interface can be displayed in a portion of the current interface of the electronic device, or it can be a different interface from the current interface of the electronic device.

[0403] In some embodiments, the above settings interface may include a smart assistant floating ball control (such as...) Figure 1G The notification control 107 in the system is used to remind the user that the smart assistant has been started and to trigger the smart assistant to end voice recognition.

[0404] This application does not limit the display position, display style, display size, or other parameters of the Smart Assistant floating ball control.

[0405] Therefore, after the electronic device displays the aforementioned user interface, the user can confirm that the smart assistant has been activated.

[0406] In this embodiment, after the preset wake-up module of the smart assistant is activated, it can continuously run on the ADSP, detecting the received data in real time and determining whether the received data is suitable for waking up the smart assistant. This ensures timely activation of the smart assistant when the received data matches the criteria. The ADSP can trigger a breath wake-up event. Therefore, the ADSP can begin reporting breath wake-up events hierarchically, controlling the smart assistant's activation via the audio trigger module, ensuring both convenience and timeliness in activating the smart assistant.

[0407] As can be seen, the above-mentioned breath-activated smart assistant technology does not require a wake word or trigger button, simplifying the voice interaction process and enabling the smart assistant to be launched more quickly.

[0408] It should be noted that the above steps S118-S119 are only used as an example of the Android architecture to illustrate the internal implementation, and do not mean that the embodiments of this application are limited to this. Optionally, the above steps S118-S119 can be implemented by one or more internal modules / devices of a system. For example, they can all be executed by an application at the application layer. Alternatively, S118-S119 can be implemented by the main process of the system, or by the underlying driver. It can be a Windows system or an iOS system, etc. This application does not limit this.

[0409] As described above, in step S12, the electronic device detects breath wake-up. The following example uses a preset wake-up module deployed in the Audio Digital Signal Processor (ADSP) to specifically describe the process of breath wake-up detection implemented by the ADSP. It can be understood that the preset wake-up module deployed in the second processor can also refer to this method. Figure 8 Perform breath-awakening detection.

[0410] like Figure 8 As shown, combined with Figure 4 and Figure 5 The corresponding embodiments, taking the example of an audio data access module being independent of a preset wake-up module, with both the audio data access module and the preset wake-up module being deployed on an audio digital signal processor (ADSP), exemplify the process of ADSP implementing breath wake-up recognition.

[0411] Step S201: The preset wake-up module starts the Qsocket registration for the hand-raise event.

[0412] Optionally, combined Figure 4 and Figure 5 In a corresponding embodiment, a QTcpSocket can be started by a preset wake-up module to register and handle the hand-raising event, and the "hand-raising event" can be listened to and handled through a socket (such as QTcpSocket or QUdpSocket).

[0413] It is understandable that the preset wake-up module can also register a raise-to-hand event by starting QMI, and this application does not make specific limitations on this.

[0414] Step S202: The real-time operating system registers the hand-raise event.

[0415] Optionally, the Real-Time Operating System (RTOS) responds to the preset wake-up module's initiation of the Qsocket raise event registration by registering the raise event with the sensor server. Specifically, the RTOS calls a registration function to register the raise event, which may include an event ID, callback function, etc. The return value may be a status code indicating whether the registration was successful.

[0416] Step S203: The preset wake-up module obtains the return value.

[0417] For example, after the real-time operating system calls the registration function, the sensor server returns the return value to the preset wake-up module via the RTOS. The preset wake-up module obtains the return value and checks it to determine whether the registration of the hand-raise event was successful.

[0418] Step S204: The preset wake-up module modifies the flag bit of the audio data access module to disable the audio data buffer.

[0419] Optionally, combined Figure 4 and Figure 5 In a corresponding embodiment, upon successful registration, the preset wake-up module modifies the flag bit of the audio data access module to disable the audio data cache. Optionally, the breath-activated wake-up pre-module of the preset wake-up module can control the audio data access module to disable the data cache by modifying the flag bit of the audio data access module.

[0420] For example, the breath wake-up pre-module (notification unit) modifies the flag of the audio data access module, setting the flag to a first value such as TRUE, to control the audio data access module to disable audio data buffering. For instance, if the flag of the audio data access module is flow_ctrl_cfg.is_need_mock_read, the notification unit sets flow_ctrl_cfg.is_need_mock_read to TRUE to modify the flag of the audio data access module, and passes the modified flag to the audio data access module through the capi_freewakeup_send_ctrl_msg_flow_to_dam function. When the audio data access module detects that the flag is TRUE, it disables audio data buffering and does not perform audio data buffering.

[0421] In some embodiments, during step S116 above, when the preset wake-up module running on the ADSP performs breath wake-up recognition, the preset wake-up module controls the audio data access module to close the data cache.

[0422] Step S205: Control the ADSP to enter low-power mode.

[0423] Combination Figure 4 and Figure 5 In a corresponding embodiment, after the preset wake-up module controls the audio data access module to close the data cache, the preset wake-up module controls the ADSP to enter a low-power mode.

[0424] For example, the preset wake-up module or its detection and processing module calls a function to notify the ADSP to enter low-power mode. For instance, the function calls the function to notify the aforementioned low-power island to enter low-power mode.

[0425] Step S206: The preset wake-up module detects the hand-raise event.

[0426] Combination Figure 4 and Figure 5 In a corresponding embodiment, when the ADSP is in low-power mode, the ADSP runs a gesture detection module and a breath wake-up module deployed in the ADSP's low-power memory space. The gesture detection module detects hand-raising events.

[0427] For example, the sensor driver acquires data from an accelerometer, gyroscope, or IMU, and transmits this data as motion data to the gesture detection module. The gesture detection module periodically checks (polulates) whether the electronic device is in a raised state based on this motion data. When the gesture detection module detects that the electronic device is in a raised state, it detects a hand-raising event. The details regarding motion data and the determination of whether the electronic device is in a raised state or has experienced a hand-raising event are as described above and will not be repeated here.

[0428] Step S207: The sensor server reports the hand-raise event.

[0429] For example, combined Figure 4 and Figure 5 In a corresponding embodiment, when the gesture detection module detects a hand-raising event or whether the electronic device is in a raised state, the sensor server reports the hand-raising event to the RTOS. The RTOS can then report the hand-raising event to the preset wake-up module through the callback function in the registration function.

[0430] Step S208: Control the ADSP to exit low-power mode.

[0431] For example, combined Figure 4 and Figure 5 In a corresponding embodiment, the preset wake-up module responds to the hand-raise event and controls the ADSP to exit low-power mode. After the ADSP exits low-power mode, it sends the hand-raise state to the breath wake-up startup module through the state forwarding module deployed in the non-low-power memory space of the ADSP.

[0432] Optionally, the breath wake-up activation module activates the breath wake-up pre-module in response to the electronic device's motion state being lifted. The exit low-power unit in the breath wake-up pre-module can control the ADSP to exit low-power mode. The motion data acquisition unit, notification unit, and activation unit, deployed in the non-low-power memory space, then operate.

[0433] For example, the preset wake-up module or breath wake-up activation module exits the low-power unit and calls a function to notify the ADSP to enter low-power mode. For instance, the function calls a function to notify the aforementioned low-power island to enter low-power mode.

[0434] Step S209: The preset wake-up module modifies the flag bit of the audio data access module to perform audio data caching.

[0435] For example, combined Figure 4 and Figure 5 In a corresponding embodiment, the ADSP operates the notification unit of the breath wake-up pre-module. The notification unit modifies the flag bit of the audio data access module, assigning the flag of the audio data access module a second value, such as FALSE, to control the audio data access module to perform audio data buffering. When the audio data access module detects that the flag is the second value, it stores the speech captured by the audio acquisition device, such as a microphone, including either the first speech or the second speech.

[0436] For example, the notification unit of the breath wake-up preamp module modifies the flag bit of the audio data access module by assigning it the second value, FALSE, for audio data caching. Specifically, the flag bit of the audio data access module is `flow_ctrl_cfg.is_need_mock_read`. The notification unit assigns `flow_ctrl_cfg.is_need_mock_read` the second value, FALSE, to modify the audio data access module's flag bit and passes the modified flag bit to the audio data access module through the `capi_freewakeup_send_ctrl_msg_flow_to_dam` function. When the audio data access module detects that the flag bit is the second value, FALSE, it performs audio data caching, that is, it acquires the speech captured by the microphone and caches it in the buffer.

[0437] Step S210: The preset wake-up module determines whether the breath wake-up event is triggered.

[0438] For example, combined Figure 4 and Figure 5In the corresponding embodiment, the motion data acquisition unit and the startup unit of the breath wake-up pre-module operate. The startup unit activates the breath wake-up detection module. After startup, the breath wake-up detection module acquires voice data. The motion data acquisition unit transmits the acquired motion data to the breath wake-up detection module. The breath wake-up detection module can perform breath wake-up recognition based on the acquired voice and motion data. The breath wake-up detection module can determine whether a breath wake-up event has been triggered based on the acquired voice and / or motion data. The relevant content regarding triggering a breath wake-up event can be found above and will not be repeated here.

[0439] If the breath wake-up pre-module detects a breath wake-up event, the breath wake-up event is reported via ADSP in S117.

[0440] Step S211: When it is determined that the breath wake-up event has been triggered, the preset wake-up module wakes up the application processor.

[0441] For example, combined Figure 4 and Figure 5 In a corresponding embodiment, the preset wake-up module determines that a breath wake-up event has been triggered and wakes up the application processor. When the breath wake-up detection module detects the acquired audio data and / or motion data and determines that a breath wake-up event has been triggered, the ADSP wakes up the application processor (AP). The relevant details regarding triggering a breath wake-up event can be found above and will not be repeated here.

[0442] like Figure 1F As shown, after the phone screen is off, the user can lift the phone, bring the bottom of the phone close to their mouth, and speak into the microphone 2 at the bottom. The phone responds to the input, including lifting the phone, bringing the bottom of the phone close to the mouth, and speaking into the microphone 2 at the bottom, waking up the AP and restoring the smart assistant through the AP, displaying as shown. Figure 1G The interface shown, in some embodiments, displays... Figure 1G The interface shown can also be displayed afterward. Figure 1H The interface shown.

[0443] Step S212: The preset wake-up module modifies the flag bit of the audio data access module to disable the audio data buffer.

[0444] For example, combined Figure 4 and Figure 5 In a corresponding embodiment, the preset wake-up module determines that a breath wake-up event has been triggered, and modifies the flag bit of the audio data access module to disable the audio data cache.

[0445] For example, in response to a breath-wake event, the detection and processing module modifies the flag of the audio data access module, setting it to the first value TRUE, thereby controlling the audio data access module to disable its data buffer. The specific implementation of the detection and processing module's modification of the audio data access module's flag can be found in the notification unit and will not be elaborated upon here.

[0446] Step S213: When it is determined that the breath wake-up event has not been triggered, the preset wake-up module modifies the flag bit of the audio data access module to disable the audio data cache.

[0447] For example, combined Figure 4 and Figure 5 In a corresponding embodiment, if the preset wake-up module determines that the breath wake-up event has not been triggered, the preset wake-up module modifies the flag bit of the audio data access module to disable the audio data cache.

[0448] For example, in response to the absence of a breath wake-up event, the detection and processing module controls the audio data access module to disable data buffering by modifying the flag bit of the audio data access module and setting the flag bit of the audio data access module to the first value TRUE.

[0449] Step S214: Control the ADSP to enter low-power mode.

[0450] For example, combined Figure 4 and Figure 5 In a corresponding embodiment, the preset wake-up module determines that no breath wake-up event has been triggered and controls the ADSP to enter a low-power mode.

[0451] For example, in response to the absence of a breath wake-up event, the detection processing module controls the ADSP to enter a low-power mode, and the ADSP continues to run the preset wake-up module. That is, when detection fails, the ADSP re-executes from step S204.

[0452] It should be noted that the above steps S201, S203 to S214 are only used as an example of the preset wake-up module of ADSP to illustrate the internal implementation. It can be understood that the above steps S201, S203 to S214 can be implemented by one or more modules inside ADSP; the above steps S201 to S214 can be implemented by one or more modules inside the second processor.

[0453] speech recognition stage

[0454] As described above, in step S13, when a breath wake-up event is detected, the electronic device performs voice recognition. The following example illustrates the specific process of voice recognition performed by the electronic device, that is, the optional implementation of step S13, such as... Figure 9As shown, the recognition stage of the voice interaction method provided in the embodiments of this application is illustrated by way of example.

[0455] In step S301, the smart assistant fails to call the audio strategy service module and does not start speech recognition.

[0456] In this embodiment, after the smart assistant is started, it calls the audio policy service module. If the call to the audio policy service module is successful, the channel between the smart assistant and the audio policy service module is opened. If the call fails, the channel between the smart assistant and the audio policy service module is closed, and the smart assistant does not initiate speech recognition. For example, the smart assistant does not send a request to the audio policy service module to initiate speech recognition.

[0457] In step S301, the "×" between the smart assistant and the audio policy service module indicates that the smart assistant failed to call the audio policy service module.

[0458] It should be noted that the smart assistant's call to the audio strategy service module may result in either success or failure. The smart assistant will fail to call the audio strategy service module if the audio channel of the audio strategy service module is occupied, or if the smart assistant exits after starting successfully.

[0459] For example, after the smart assistant is launched, the recording function of the electronic device is enabled, the audio channel of the audio policy service module is occupied, and the smart assistant cannot successfully call the sound policy service module.

[0460] Step S302: The audio trigger module of the framework layer times out and starts breath wake-up recognition.

[0461] In this embodiment, when the smart assistant fails to call the audio strategy service module, the audio triggering module of the framework layer initiates breath wake-up recognition. For example, the audio triggering module of the framework layer sends a request to initiate breath wake-up recognition through the hardware abstraction layer.

[0462] In this embodiment, when the smart assistant successfully calls the audio strategy service module, it sends a request to initiate speech recognition to the audio strategy service module. In response to this request, the audio strategy service module sends a request to the audio trigger module in the framework layer to initiate breath wake-up recognition. The audio trigger module in the framework layer starts a timer to determine whether a request to initiate breath wake-up recognition has been received within the timer's set time. If no request to initiate breath wake-up recognition is received within the set time, the smart assistant's call to the audio strategy service module has failed. If no request to initiate breath wake-up recognition is received within the set time, the listening timeout occurs.

[0463] In some embodiments, the timer's timing interval is 1-5 seconds. For example, the timer's timing interval is 3 seconds.

[0464] Step S311: The smart assistant successfully calls the audio strategy service module and starts voice recognition.

[0465] In this embodiment, the smart assistant initiates speech recognition upon successfully calling the audio policy service module. For example, the smart assistant sends a request to the audio policy service module to initiate speech recognition.

[0466] Step S312: The audio strategy service module starts speech recognition.

[0467] In this embodiment, the audio policy service module initiates speech recognition upon successful invocation. For example, in response to the intelligent assistant's request to initiate speech recognition, the audio policy service module sends a request to initiate speech recognition to the audio triggering module in the framework layer.

[0468] In step S313, the audio trigger module of the framework layer stops breath wake-up recognition.

[0469] In this embodiment, the audio triggering module of the framework layer, in response to the audio policy service module's request to start speech recognition, controls the ADSP to stop breath wake-up recognition. For example, in response to the request to start speech recognition sent by the audio policy service module, the audio triggering module of the framework layer sends a request to stop breath wake-up recognition to the hardware abstraction layer. The hardware abstraction layer, in response to the received request to stop breath wake-up recognition, sends a request to stop breath wake-up recognition to the ADSP.

[0470] For example, the audio triggering module of the framework layer can send a request to the hardware abstraction layer to stop breath wake-up recognition through the function StopRecoginition.

[0471] In some embodiments, the audio triggering module of the framework layer can send a request to the hardware abstraction layer to stop breath wake-up recognition after receiving a breath wake-up event. In response to the request, the hardware abstraction layer controls the preset wake-up module in the ADSP to stop operating. Since the breath wake-up event has been triggered, it can be considered as consuming one breath wake-up event, and the preset wake-up module can stop or pause its operation.

[0472] Step S314: The hardware abstraction layer acquires the voice.

[0473] In this embodiment, the hardware abstraction layer obtains speech, such as a first speech and / or a second speech, from the audio data access module. For example, the hardware abstraction layer obtains speech from the audio data access module in response to a request to stop breath wake-up recognition.

[0474] In some embodiments, the audio policy service module can transfer the first speech and / or the second speech from the audio data access module via the audio source 1999 stream.

[0475] In step S315, the audio policy service module sends a voice message to the smart assistant.

[0476] In this embodiment, after the audio policy service module retrieves the voice stored in the audio data access module from the hardware abstraction layer, it sends the voice to the smart assistant so that the smart assistant can perform voice recognition on the voice.

[0477] Step S316: The smart assistant performs voice recognition.

[0478] In this embodiment, the smart assistant performs speech recognition on the voice sent by the audio policy service module. The smart assistant not only recognizes the meaning of the audio data but also converts it into text.

[0479] In step S317, the audio strategy service module determines that speech recognition has ended and then stops speech recognition.

[0480] Among them, the intelligent assistant can use a variety of methods to determine the end of voice recognition based on audio data.

[0481] In some embodiments, when the smart assistant receives a user's action on the smart assistant's floating ball control, it can determine that the user wants to end voice recognition. Therefore, the smart assistant can respond to the action on the smart assistant's floating ball control and end voice recognition.

[0482] In other embodiments, the intelligent assistant determines that speech recognition has ended when it determines that the received audio data corresponds to a complete word or sentence.

[0483] Suppose the audio data is "How's the weather today?". Upon receiving "How's the weather today?", the smart assistant can determine that it is a complete sentence, and thus the speech recognition process is complete.

[0484] In this embodiment of the application, when the smart assistant determines that the speech recognition has ended, it sends a request to the audio policy service module to stop speech recognition.

[0485] In some embodiments, the smart assistant displays the speech recognition result when it determines that the speech recognition is complete.

[0486] The intelligent assistant can convert audio data into text format during the audio data recognition process and display it. Figure 1G or Figure 1H On the interface.

[0487] For example, such as Figure 1G As shown, when the audio data is "How is the weather today?", the smart assistant can display the corresponding text data on the user interface after recognizing "How is the weather today?".

[0488] In other embodiments, the smart assistant can display the corresponding response to the audio data on the user interface after the audio data recognition is completed.

[0489] For example, such as Figure 1H As shown, when the audio data is "How is the weather today?", after recognizing "How is the weather today?", the smart assistant can determine that the user wants to know today's weather conditions and can display the current temperature, maximum temperature, and minimum temperature on the user interface.

[0490] In this embodiment, the smart assistant stops speech recognition when it determines that the speech recognition has ended. For example, upon determining that speech recognition has ended, it sends a request to stop speech recognition to the audio policy service module.

[0491] In step S318, the audio policy service module stops speech recognition.

[0492] In this embodiment of the application, the audio policy service module responds to the request to stop speech recognition sent by the smart assistant by sending a request to stop speech recognition to the audio triggering module of the framework layer.

[0493] Step S319: The audio strategy service module initiates breath wake-up recognition.

[0494] In this embodiment, the audio triggering module of the framework layer initiates breath wake-up recognition in response to a request to stop speech recognition. For example, the audio triggering module of the framework layer sends a request to the hardware abstraction layer to initiate breath wake-up recognition in response to a request to stop speech recognition. The hardware abstraction layer then sends the request to the ADSP to initiate breath wake-up recognition.

[0495] In step S320, the hardware abstraction layer stops acquiring voice.

[0496] In this embodiment, in response to the request to initiate breath wake-up recognition, the hardware abstraction layer stops transferring speech from the audio data access module, that is, stops obtaining the speech stored in the audio data access module.

[0497] Step S321: ADSP initiates breath wake-up recognition.

[0498] In this embodiment, the ADSP responds to the request to initiate breath wake-up recognition by performing breath wake-up recognition, as detailed in the following reference. Figure 8 This will not be elaborated upon here.

[0499] When the smart assistant successfully calls the audio strategy service module, the audio triggering module in the framework layer can send a request to the hardware abstraction layer to stop the breath wake-up module. The hardware abstraction layer then controls the ADSP's preset wake-up module to stop running, thus avoiding interruption of the current speech recognition.

[0500] The audio triggering module in the framework layer can send a request to the hardware abstraction layer to start breath wake-up recognition when the smart assistant determines that the speech recognition has ended. The hardware abstraction layer controls the ADSP's preset wake-up module to start running, which makes it easy to restore the function of the smart assistant's preset wake-up module.

[0501] Regardless of whether the smart assistant's call to the audio strategy service module is successful or not, the audio triggering module can send a request to the hardware abstraction layer to start breath wake-up recognition. The hardware abstraction layer controls the preset wake-up module to start running, making it easy to restore the function of the smart assistant's preset wake-up module.

[0502] In other words, the audio triggering module can send a request to the hardware abstraction layer to start breath wake-up recognition at the appropriate time. The hardware abstraction layer controls the preset wake-up module to start running, which can not only avoid the interruption of the current voice recognition, but also ensure that the smart assistant can be woken up normally next time.

[0503] It should be noted that the above steps S301 to S321 are used as examples of the intelligent assistant, the audio trigger module of the framework layer, the audio policy service module, the hardware abstraction layer and the ADSP to illustrate the internal implementation. It can be understood that the above steps S401 to S406 can be implemented by the first processor, or by a certain application or the system main process. This application embodiment does not make specific limitations on this.

[0504] Dialogue phase

[0505] like Figure 10 The diagram illustrates the dialogue phase of a voice interaction method. This voice interaction method is applied to an electronic device, which includes an audio service, a hardware abstraction layer, and an ADSP.

[0506] Step S401: The Audio service confirms successful voiceprint recognition and starts recording the conversation.

[0507] In this embodiment, after waking the electronic device using a wake-word-free method, voiceprint recognition is automatically performed on the user-input audio data. After verifying the user's legitimacy based on voiceprint recognition, the device automatically unlocks and begins recording the conversation. This achieves voice-controlled unlocking of the electronic device, i.e., contactless unlocking.

[0508] Step S402: The Audio service opens the audio input stream.

[0509] In this embodiment, the audio policy service module opens the audio input stream upon successful voiceprint recognition. For example, upon successful voiceprint recognition, the audio policy service module sends a request to the hardware abstraction layer to open the audio input stream. Sending the request to the hardware abstraction layer may include: the audio policy service module initializing the audio input stream and concurrently returning a handle to read audio data.

[0510] For example, the audio policy service module can send a request to the hardware abstraction layer to open the audio input stream through the function adev_open_input_stream()audio_io_handle_thandle.

[0511] The adev_open_input_stream() function initializes an audio input stream and returns a handle of type audio_io_handle_thandle, which can be used to read audio data, control the audio input stream, and configure audio parameters.

[0512] Step S403: The Hardware Abstraction Layer creates an audio stream.

[0513] In this embodiment of the application, the hardware abstraction layer creates an audio stream in response to a request to open an audio stream.

[0514] For example, an audio stream can be created using the function CreateStreamln(), and a class StreamlnPrimary() can be created within the CreateStreamln() method.

[0515] Step S404: The Audio service starts reading audio data.

[0516] In this embodiment, the audio policy service module initiates the reading of audio data. For example, the audio policy service module sends a request to the hardware abstraction layer to initiate the reading of audio data.

[0517] For example, the audio policy service module can send a request to the hardware abstraction layer to initiate the reading of audio data through the in_read() function.

[0518] Step S405: The hardware abstraction layer reads the audio data.

[0519] In this embodiment, the hardware abstraction layer controls the ADSP to read audio data. For example, in response to a request from the audio policy service module to initiate audio data reading, the hardware abstraction layer sends a request to the ADSP to read audio data.

[0520] In step S406, the ADSP reads audio data from the audio acquisition device in real time.

[0521] In this embodiment, the ADSP can read audio data in real time from an audio acquisition device such as a microphone and process the audio data using corresponding algorithms. For example, the ADSP reads audio data in real time from an audio acquisition device such as a microphone in response to a request for audio data.

[0522] It should be noted that the above steps S401 to S406 are illustrated using Audio service, hardware abstraction layer and ADSP as examples to illustrate the internal implementation. It can be understood that the above steps S401 to S406 can be implemented by the first processor, or by a certain application or system main process. This application embodiment does not specifically limit this.

[0523] Please see Figure 11 This application provides an example of another voice interaction method. This voice interaction method can be applied to the aforementioned electronic device, which includes a first microphone and a second microphone.

[0524] In step S501, at the first moment, the electronic device is lifted and the screen of the electronic device is in a turned-off state. The power consumption of the electronic device changes from a first value to a second value. Before the first moment, the screen of the electronic device is in a turned-off state, the electronic device is not lifted, and the second value is greater than the first value.

[0525] It can be understood that an electronic device being lifted means that the electronic device is in a lifted state, or that an electronic device is lifted.

[0526] It can be understood that when the screen of an electronic device is off, the electronic device is screen-off.

[0527] In step S502, between the first moment and the second moment, the electronic device receives the first voice.

[0528] It is understandable that electronic devices can collect voice data through audio acquisition devices, such as... Figure 1F The electronic device was lifted up, and it received the first voice message, "How's the weather today?"

[0529] In step S503, at the second moment, the electronic device turns on its screen. After the second moment, the power consumption of the electronic device is a third value, which is greater than the second value.

[0530] For example, combined Figures 6A to 6CIn a corresponding embodiment, before time T1 or T11, the screen of the electronic device is off, the electronic device is not lifted, and the power consumption of the electronic device is a first value or within a first power consumption range. At time T1 or T11, the electronic device is lifted, and the screen is off; the power consumption of the electronic device changes from the first value to a second value, or the power consumption of the electronic device falls within a second power consumption range. The second value is greater than the first value, and any power consumption within the first power consumption range is less than any power consumption within the second power consumption range. Figure 6C Between T11 and T12, the electronic device receives the first voice message. At time T12, the electronic device determines that a breath wake-up event has been triggered, and the screen of the electronic device turns on. After the second time point, the power consumption of the electronic device is a third value, which is greater than the second value.

[0531] In some embodiments, after the second time point, the electronic device screen is turned off. At the third time point, the electronic device is lifted, and the screen is off. The power consumption of the electronic device changes from a first value to a third value. Before the third time point, the screen is off, the electronic device is not lifted, and the third value is greater than the first value. Between the third and fourth time points, the electronic device does not receive voice input. At the fourth time point, the power consumption of the electronic device is the first value.

[0532] It is understood that the third value mentioned above can be the same as or close to the second value. The third value and the second value can be understood as the actual power consumption of the electronic device fluctuating within a reasonable range.

[0533] It is understandable that both the third and second values ​​fall within the second power consumption range.

[0534] For example, after the second moment, the screen of the electronic device may be turned off in the following situations, including but not limited to: Situation 1, such as Figure 6B As shown, between T11 and T12, the electronic device receives the first voice message. At time T12, the electronic device determines that the breath wake-up event has not been triggered, and the electronic device remains in an off state, i.e., the screen is off. Scenario 2: At time T12, the electronic device determines that the breath wake-up event has been triggered, and the screen turns on. After the screen turns on, voice recognition is performed. The electronic device determines that voice recognition is complete, and the screen turns off.

[0535] After the second moment, the screen of the electronic device is turned off, that is, before the third moment, the screen of the electronic device is in an off state, the electronic device is not lifted, the power consumption of the electronic device is the first value, or the power consumption of the electronic device is within the first power consumption range.

[0536] For example, combined Figures 6A to 6CIn a corresponding embodiment, at time T1 or T11, the electronic device is lifted, and the screen of the electronic device is off, causing the power consumption of the electronic device to change from a first value to a second value. Figure 6B For example, if no voice is received between T11 and T12, then... Figure 6B As shown, the electronic device did not receive the first voice message between T11 and T12. Based on the fact that the first voice message was not received between T11 and T12, it was determined that the breath wake-up event was not triggered. Therefore, in or after T12, the power consumption of the electronic device is the first value, or the power consumption of the electronic device is within the first power consumption range.

[0537] In some embodiments, after the second time point, the electronic device screen is turned off. At the fifth time point, the electronic device is lifted, and the screen of the electronic device is off. The power consumption of the electronic device changes from a first value to a fourth value. Before the fifth time point, the screen of the electronic device is off, the electronic device is not lifted, and the fourth value is greater than the first value. Between the fifth and sixth time points, the electronic device receives a second voice message. The first distance between the sound source of the second voice message and the second microphone is greater than or equal to the second distance between the sound source of the second voice message and the first microphone. At the sixth time point, the power consumption of the electronic device is the first value.

[0538] It is understood that the fourth value mentioned above may be the same as or close to the third or second value. The second, third and fourth values ​​can be understood as the actual power consumption of the electronic device fluctuating within a reasonable range.

[0539] It is understandable that the second, third, and fourth values ​​all fall within the second power consumption range.

[0540] After the second moment, the screen of an electronic device may turn off in the above-mentioned circumstances, which will not be elaborated here.

[0541] After the second moment, the screen of the electronic device is turned off. That is, before the fifth moment, the screen of the electronic device is in an off state, the electronic device is not lifted, and the power consumption of the electronic device is the first value, or the power consumption of the electronic device is within the first power consumption range.

[0542] For example, combined Figures 6A to 6C In a corresponding embodiment, at time T1 or T11, the electronic device is lifted, and the screen of the electronic device is off, causing the power consumption of the electronic device to change from a first value to a second value. Figure 6B Taking the reception of voice between T11 and T12 as an example, Figure 6BAs shown, between T11 and T12, the electronic device receives the first voice. Based on the fact that between the fifth and sixth moments, the electronic device receives the second voice, and the first distance between the sound source of the second voice and the second microphone is greater than or equal to the second distance between the sound source of the second voice and the first microphone, the electronic device determines that the breath wake-up event has not been triggered. Therefore, at or after T12, the power consumption of the electronic device is the first value, or the power consumption of the electronic device is within the first power consumption range.

[0543] In some embodiments, between a first moment and a second moment, the electronic device receives a first voice message, and a third distance between the sound source of the first voice message and a second microphone is less than a fourth distance between the sound source of the first voice message and the first microphone. At the second moment, the electronic device illuminates its screen, including: based on the third distance being less than the fourth distance, the electronic device illuminates its screen at the second moment; and displays a first interface related to the first voice message.

[0544] The first microphone, second microphone, third distance, fourth distance, first interface, and first voice can be referred to above and will not be repeated here.

[0545] In some embodiments, at a second moment, the electronic device turns on its screen, including: based on a first voice including a human voice, the electronic device turns on its screen at a second moment; and displays a first interface, the first interface being related to the first voice.

[0546] The human voice, the first interface, and the first voice are described above and will not be repeated here.

[0547] In some embodiments, prior to the first moment, a second interface is further displayed. The second interface includes a first control, which is in a first state. In response to an operation on the first control, the first control is in a second state, and the first control in the second state instructs the electronic device to wake up the voice interaction module without a wake-word.

[0548] The second interface, first control, first state, second state, and audio interaction module can be referenced above and will not be repeated here.

[0549] In some embodiments, before a first moment, the screen of the electronic device is off and the electronic device is not lifted, including: before the first moment, the screen of the electronic device is off, the electronic device is not lifted, the first processor of the electronic device is in a sleep state, and the power consumption of the electronic device is a first value. At the first moment, the electronic device is lifted, and the screen of the electronic device is off, and the power consumption of the electronic device changes from the first value to a second value, including: at the first moment, the electronic device is lifted, the screen of the electronic device is off, the first processor remains in a sleep state, the operating voltage and / or operating current of the first memory of the second processor of the electronic device increases, and based on the increase in the operating voltage and / or operating current of the first memory, the power consumption of the electronic device changes from the first value to the second value.

[0550] In some embodiments, before a first moment, when the screen of the electronic device is off and the electronic device is not lifted, the following includes: before the first moment, when the screen of the electronic device is off and the electronic device is not lifted, the first processor of the electronic device is in a sleep state, and the second processor of the electronic device is in a low-power mode; based on the first processor being in a sleep state and the second processor being in a low-power mode, the power consumption of the electronic device is a first value. When the electronic device is lifted and the screen of the electronic device is off at the first moment, the power consumption of the electronic device changes from the first value to a second value, including: when the screen of the electronic device is off and the first processor is in a sleep state, in response to the electronic device being lifted, the second processor exits the low-power mode; based on the second processor exiting the low-power mode, the power consumption of the electronic device changes from the first value to the second value.

[0551] In some embodiments, the second processor being in a low-power mode includes: the first memory of the second processor being in a low-power state, the operating voltage of the first memory being a first voltage, and the operating current of the first memory being a first current. The second processor exiting the low-power mode includes: the first memory being in a non-low-power state, the operating voltage of the first memory being a second voltage, and the operating current of the first memory becoming a second current, wherein the second voltage is greater than the first voltage, and / or the second current is greater than the first current.

[0552] It can be understood that a processor's memory can be considered a storage unit or component closely related to the processor, used for temporarily or permanently storing data and programs. For example, the first memory of a second processor is a storage unit or component related to the second processor. The first memory may house a portion of a pre-set wake-up module; details can be found in [reference needed]. Figure 4 and Figure 5 .

[0553] In some embodiments, at the second moment, the electronic device turning on the screen includes: at the second moment, the first processor of the electronic device is in a working state, the electronic device turns on the screen, and based on the first processor being in a working state, the power consumption of the electronic device is a third value.

[0554] In some embodiments, before the first moment, the method further includes: controlling a second processor of the electronic device to enter a low-power mode; and assigning a first value to a flag bit of the audio data cache to disable the audio data cache.

[0555] Optionally, the flag of the audio data cache is assigned the first value, that is, the flag of the audio data access module is assigned the first value.

[0556] In some embodiments, the first moment when the electronic device is lifted further includes: detecting a hand-lifting event, determining that the electronic device is lifted in response to the hand-lifting event, controlling the second processor to exit the low-power mode; and assigning a second value to the flag bit of the audio data cache to open the audio data cache.

[0557] Optionally, the flag of the audio data cache is assigned a second value, that is, the flag of the audio data access module is assigned a second value.

[0558] For example, combined Figure 8 For specific implementation examples, please refer to steps S207 to S209, which will not be repeated here.

[0559] In some embodiments, between the first time and the second time, the electronic device receiving the first voice includes: between the first time and the second time, the electronic device receiving the first voice and storing the first voice in an audio data cache, the first microphone collecting the signal strength of the first voice as a first signal strength, the second microphone collecting the signal strength of the first voice as a second signal strength, and the difference between the second signal strength and the first signal strength as a first difference; determining whether the first difference is greater than a first threshold, whether the second signal strength is greater than a second threshold, and whether the first voice includes human voice; at the second time, the electronic device turning on the screen includes: at the second time, based on the first difference being greater than the first threshold, the second signal strength being greater than the second threshold, and the first voice including human voice, controlling the first processor of the electronic device to exit the sleep state, and assigning the flag bit of the audio data cache to a first value to close the audio data cache, and the electronic device turning on the screen.

[0560] In some embodiments, the electronic device not receiving voice between the third and fourth time points includes: between the third and fourth time points, in response to not receiving voice, the electronic device controls the second processor to enter a low-power mode and assigns a first value to the flag bit of the audio data cache to disable the audio data cache.

[0561] In some embodiments, between a fifth time and a sixth time, the electronic device receives a second voice message, and a third distance between the sound source of the second voice message and the second microphone is greater than or equal to a fourth distance between the sound source of the second voice message and the first microphone. This includes: between the fifth time and the sixth time, the electronic device receives the second voice message; the first microphone acquires a signal strength of the second voice message as a third signal strength, the second microphone acquires a signal strength of the second voice message as a fourth signal strength, and the difference between the fourth signal strength and the third signal strength is a second difference; based on the second difference being less than or equal to a first threshold, the second processor is controlled to enter a low-power mode, and a first value is assigned to the flag bit of the audio data cache to disable the audio data cache.

[0562] In some embodiments, the electronic device includes a voice interaction module and a preset wake-up module. A first processor of the electronic device runs the voice interaction module, and a second processor of the electronic device runs the preset wake-up module. The preset wake-up module includes a gesture detection module, a breath wake-up activation module, and a breath wake-up detection module. The gesture detection module detects the motion state of the electronic device; the breath wake-up activation module activates the breath wake-up detection module when the electronic device is lifted. The breath wake-up detection module determines whether to wake up the first processor based on received voice. The breath wake-up detection module is deployed in the first memory of the second processor. The gesture detection module and the breath wake-up activation module are deployed in the second memory of the second processor.

[0563] It can be understood that a processor's memory can be considered a storage unit or component closely related to the processor, used for temporarily or permanently storing data and programs. For example, the second memory of a second processor is a storage unit or component related to the second processor. A preset wake-up module runs on the second processor, a portion of the preset wake-up module is deployed in the first memory, and another portion of the preset wake-up module is deployed in the second memory. For details, please refer to... Figure 4 and Figure 5 This will not be elaborated upon here.

[0564] In some embodiments, before the first moment, the screen of the electronic device is off and the electronic device is not lifted. The second value being greater than the first value includes: before the first moment, the gesture detection module and the breath wake-up activation module are running on the second processor, and the breath wake-up detection module is not running on the second processor because the electronic device is not lifted; based on the gesture detection module and the breath wake-up activation module running on the second processor, but the breath wake-up detection module not running on the second processor, the first processor is in a sleep state, and the power consumption of the electronic device is the first value. At the first moment, the electronic device is lifted, and the screen of the electronic device is off. The power consumption of the electronic device changing from the first value to the second value includes: at the first moment, based on the electronic device being lifted, the gesture detection module detects a hand-raising event. The gesture detection module transmits the hand-raising event to the breath wake-up activation module. Based on the hand-raising event, the breath wake-up activation module activates the breath wake-up detection module, which runs on the second processor; based on the second processor running the breath wake-up detection module, the first processor is in a sleep state, and the power consumption of the electronic device changes from the first value to the second value.

[0565] In some embodiments, the preset wake-up module further includes a state forwarding module, which is deployed in the first memory of the second processor. At a first moment, in response to the electronic device being lifted, the gesture detection module detects a hand-lifting event and transmits the hand-lifting event to the breath wake-up initiation module, including: at the first moment, based on the electronic device being lifted, the gesture detection module detects a hand-lifting event. The gesture detection module transmits the hand-lifting event to the state forwarding module. The state forwarding module, based on the hand-lifting event, transmits a lift-up state to the breath wake-up initiation module. The breath wake-up initiation module, based on obtaining the lift-up state, activates the breath wake-up detection module.

[0566] In some embodiments, the preset wake-up module further includes a breath wake-up pre-module and an audio data access module, the audio data access module being deployed in a first memory; the breath wake-up pre-module runs on a second processor. The breath wake-up activation module, based on obtaining a lift-off state, activates the breath wake-up detection module by: triggering the breath wake-up pre-module to run; controlling the second processor to exit a low-power mode; activating the breath wake-up detection module; acquiring motion data collected by the electronic device's sensors and transmitting the motion data to the breath wake-up detection module; and assigning a second value to the flag bit of the audio data access module to control the audio data access module to perform audio data buffering.

[0567] In some embodiments, between a first moment and a second moment, the electronic device receiving the first voice includes: between the first moment and the second moment, the audio data access module, in response to a flag bit being a second value, storing the first voice collected by the sensors of the electronic device.

[0568] In some embodiments, the breath wake-up pre-module includes an exit low-power unit, a motion data acquisition unit, a notification unit, and a startup unit; the exit low-power unit is deployed in a second memory, and the motion data acquisition unit, notification unit, and startup unit are deployed in a first memory. The exit low-power unit controls the second processor to exit low-power mode. The motion data acquisition unit acquires motion data and transmits the motion data to the breath wake-up detection module. The notification unit assigns a second value to the flag bit of the audio data access module. The startup unit starts the breath wake-up detection module.

[0569] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0570] This application embodiment also provides a chip system applied to an electronic device. The chip system includes a first processor and a second processor: at a first moment, the electronic device is lifted and the screen of the electronic device is in a turned-off state. The second processor triggers the power consumption of the electronic device to change from a first value to a second value. Before the first moment, the screen of the electronic device is in a turned-off state, the electronic device is not lifted, and the second value is greater than the first value. Between the first moment and the second moment, the second processor receives first voice data. At the second moment, the second processor wakes up the first processor. After the second moment, the first processor controls the screen of the electronic device to turn on, and the power consumption of the electronic device is a third value, which is greater than the second value.

[0571] The first voice data can be understood as the voice data of the first voice mentioned above.

[0572] Optionally, after the second moment, the first processor controls the electronic device to turn off its screen; at the third moment, the electronic device is lifted and its screen is off, and the second processor triggers the power consumption of the electronic device to change from the first value to the third value. Before the third moment, the screen of the electronic device is off, the electronic device is not lifted, and the third value is greater than the first value; between the third moment and the fourth moment, the second processor does not receive voice data; at the fourth moment, the second processor triggers the power consumption of the electronic device to the first value.

[0573] Optionally, after the second moment, the electronic device screen is turned off; at the fifth moment, the electronic device is lifted, and the screen of the electronic device is off, and the second processor triggers the power consumption of the electronic device to change from the first value to the fourth value. Before the fifth moment, the screen of the electronic device is off, the electronic device is not lifted, and the fourth value is greater than the first value; between the fifth moment and the sixth moment, the second processor receives the second voice, and the first distance between the sound source of the second voice and the second microphone is greater than or equal to the second distance between the sound source of the second voice and the first microphone; at the sixth moment, the second processor triggers the power consumption of the electronic device to change to the first value.

[0574] Optionally, between the first and second moments, the second processor receives the first voice data, and the third distance between the sound source of the first voice data and the second microphone is less than the fourth distance between the sound source of the first voice data and the first microphone; at the second moment, the electronic device lights up its screen, including: based on the third distance being less than the fourth distance, the first processor controls the electronic device to light up its screen at the second moment; and displays a first interface, the first interface including the content of the first voice.

[0575] Optionally, at a second moment, the electronic device lights up its screen, including: based on first voice data including human voice, a first processor controls the electronic device to light up its screen at the second moment; and displays a first interface, the first interface including the content of the first voice.

[0576] The chip system provided in this application embodiment can implement the voice interaction methods in the above-described method embodiments.

[0577] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the voice interaction method in the above method embodiments.

[0578] This application also provides a computer storage medium including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the voice interaction method as described in the above embodiments.

[0579] In this application, the electronic devices, computer storage media, computer program products, or chip systems provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0580] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0581] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0582] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0583] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0584] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0585] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A voice interaction method applied to electronic devices, characterized in that, The method includes: At the first moment, the electronic device is lifted and the screen of the electronic device is off. The power consumption of the electronic device changes from a first value to a second value. Before the first moment, the screen of the electronic device is off and the electronic device is not lifted. The second value is greater than the first value. Between the first and second moments, the electronic device receives the first voice message; At the second moment, the screen of the electronic device is turned on. After the second moment, the power consumption of the electronic device is a third value, which is greater than the second value. After the second moment, the electronic device screen turns off; At the third moment, the electronic device is lifted and the screen of the electronic device is off. The power consumption of the electronic device changes from the first value to the third value. Before the third moment, the screen of the electronic device is off and the electronic device is not lifted. The third value is greater than the first value. Between the third and fourth moments, the electronic device did not receive any voice. At the fourth moment, the power consumption of the electronic device is the first value.

2. The voice interaction method as described in claim 1, characterized in that, The electronic device includes a first microphone and a second microphone; the method further includes: At the fifth moment, the electronic device is lifted and the screen of the electronic device is off. The power consumption of the electronic device changes from the first value to the fourth value. Before the fifth moment, the screen of the electronic device is off and the electronic device is not lifted. The fourth value is greater than the first value. Between the fifth and sixth moments, the electronic device receives a second voice message, wherein a first distance between the sound source of the second voice message and the second microphone is greater than or equal to a second distance between the sound source of the second voice message and the first microphone; At the sixth moment, the power consumption of the electronic device is the first value.

3. The voice interaction method as described in claim 1 or 2, characterized in that, The electronic device includes a first microphone and a second microphone; the method further includes: Between the first moment and the second moment, the electronic device receives the first voice, and the third distance between the sound source of the first voice and the second microphone is less than the fourth distance between the sound source of the first voice and the first microphone; At the second moment, the electronic device lights up its screen, including: Based on the fact that the third distance is less than the fourth distance, the electronic device lights up its screen at the second moment; The first interface is displayed, which includes the content of the first voice message.

4. The voice interaction method as described in claim 1 or 2, characterized in that, The step of the electronic device turning on its screen at the second moment includes: Based on the fact that the first voice includes a human voice, the electronic device lights up its screen at the second moment; The first interface is displayed, which includes the content of the first voice message.

5. The voice interaction method as described in claim 1 or 2, characterized in that, Before the first moment, it also includes: A second interface is displayed, the second interface including a first control, the first control being displayed in a first state; In response to an operation on the first control, the first control is displayed in a second state, indicating that the wake-word-free wake-up function of the electronic device is enabled.

6. The voice interaction method as described in claim 1 or 2, characterized in that, The electronic device includes a first processor and a second processor, the second processor including a first memory; the step of the electronic device being in a state of being off and not being lifted before the first moment includes: Before the first moment, the screen of the electronic device is off, the electronic device is not lifted, the first processor of the electronic device is in sleep mode, and the power consumption of the electronic device is a first value. At the first moment, when the electronic device is lifted and its screen is off, the change in power consumption from a first value to a second value includes: At the first moment, the electronic device is lifted, the screen of the electronic device is turned off, the first processor remains in a sleep state, the operating voltage and / or operating current of the first memory of the second processor of the electronic device increases, and based on the increase in the operating voltage and / or operating current of the first memory, the power consumption of the electronic device changes from a first value to a second value.

7. The voice interaction method as described in claim 1 or 2, characterized in that, The electronic device includes a first processor and a second processor; the condition that the screen of the electronic device is off and the electronic device is not lifted before the first moment includes: Before the first moment, the screen of the electronic device is off, the electronic device is not lifted, the first processor of the electronic device is in sleep mode, and the second processor of the electronic device is in low-power mode. Based on the fact that the first processor is in sleep mode and the second processor is in low-power mode, the power consumption of the electronic device is a first value. At the first moment, the electronic device is lifted and its screen is off, and the power consumption of the electronic device changes from a first value to a second value, including: At the first moment, the screen of the electronic device is off and the first processor is in sleep mode. In response to the electronic device being lifted, the second processor exits the low-power mode. Based on the second processor exiting the low-power mode, the power consumption of the electronic device changes from a first value to a second value.

8. The voice interaction method as described in claim 7, characterized in that, The second processor includes a first memory; The second processor is in low-power mode including: The first memory of the second processor is in a low-power state, the operating voltage of the first memory is a first voltage, and the operating current of the first memory is a first current; The second processor exits low-power mode in the following ways: The first memory is in a non-low power state, the operating voltage of the first memory is the second voltage, and the operating current of the first memory becomes the second current, wherein the second voltage is greater than the first voltage, and / or the second current is greater than the first current.

9. The voice interaction method as described in any one of claims 1, 2, or 8, characterized in that, The electronic device includes a first processor; the step of the electronic device turning on its screen at the second moment includes: at the second moment, the first processor of the electronic device is in a working state, the electronic device turns on its screen, and based on the first processor being in a working state, the power consumption of the electronic device is a third value.

10. The voice interaction method as described in any one of claims 1, 2, or 8, characterized in that, The electronic device includes a second processor; prior to the first moment, the method further includes: The second processor of the electronic device is controlled to enter a low-power mode; Set the audio data cache flag to the first value to disable audio data caching.

11. The voice interaction method as described in claim 10, characterized in that, The lifting of the electronic device at the first moment also includes: At the first moment, a hand-raising event is detected, confirming that the electronic device has been raised; In response to the hand-raising event, control the second processor to exit low-power mode; The audio data cache flag is assigned a second value to open the audio data cache.

12. The voice interaction method as described in any one of claims 1, 2, 8, or 11, characterized in that, The electronic device includes a first processor, a first microphone, and a second microphone; the receiving of the first voice by the electronic device between the first and second moments includes: Between the first moment and the second moment, the electronic device receives the first voice and stores the first voice in the audio data buffer. The first microphone collects the signal strength of the first voice as the first signal strength, the second microphone collects the signal strength of the first voice as the second signal strength, and the difference between the second signal strength and the first signal strength is the first difference. Determine whether the first difference is greater than a first threshold, whether the second signal strength is greater than a second threshold, and whether the first speech includes human voice; The second time at which the electronic device lights up includes: At the second moment, based on the fact that the first difference is greater than the first threshold, the second signal strength is greater than the second threshold, and the first voice includes human voice, the first processor of the electronic device is controlled to exit the sleep state, and the flag bit of the audio data cache is assigned a first value to close the audio data cache, and the screen of the electronic device is turned on.

13. The voice interaction method as described in any one of claims 1, 2, 8, or 11, characterized in that, The electronic device includes a second processor; the statement that the electronic device did not receive voice between the third and fourth time points includes: Between the third and fourth moments, in response to the lack of received voice, the electronic device controls its second processor to enter a low-power mode and sets the audio data cache flag to a first value to disable the audio data cache.

14. The voice interaction method as described in claim 2, characterized in that, The electronic device includes a second processor; between the fifth and sixth moments, the electronic device receives a second voice message, wherein a third distance between the sound source of the second voice message and the second microphone is greater than or equal to a fourth distance between the sound source of the second voice message and the first microphone, including: Between the fifth and sixth moments, the electronic device receives a second voice message; the first microphone collects the signal strength of the second voice message as a third signal strength, the second microphone collects the signal strength of the second voice message as a fourth signal strength, and the difference between the fourth signal strength and the third signal strength is a second difference; based on the second difference being less than or equal to a first threshold, the second processor of the electronic device is controlled to enter a low-power mode, and the flag bit of the audio data cache is assigned a first value to disable the audio data cache.

15. A chip system applied to electronic devices, characterized in that, The chip system includes a first processor and a second processor; At the first moment, the electronic device is lifted and the screen of the electronic device is off. The second processor triggers the power consumption of the electronic device to change from a first value to a second value. Before the first moment, the screen of the electronic device is off, the electronic device is not lifted, and the second value is greater than the first value. Between the first and second moments, the second processor receives the first voice data; At the second moment, the second processor wakes up the first processor. After the second moment, the first processor controls the electronic device to turn on its screen. The power consumption of the electronic device is a third value, which is greater than the second value. After the second moment, the electronic device screen turns off; At the third moment, the electronic device is lifted and the screen of the electronic device is off. The second processor triggers the power consumption of the electronic device to change from a first value to a third value. Before the third moment, the screen of the electronic device is off and the electronic device is not lifted. The third value is greater than the first value. Between the third and fourth moments, the electronic device did not receive any voice. At the fourth moment, the second processor triggers the power consumption of the electronic device to the first value.

16. An electronic device, characterized in that, The electronic device includes a memory, a processor, a sensor, an audio acquisition device, and a display screen. The sensor is used to acquire motion data of the electronic device, the audio acquisition device is used to acquire voice of the electronic device, the display screen is used to display an interface, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory, causing the electronic device to perform the voice interaction method as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the voice interaction method as described in any one of claims 1 to 14.

Images