Voice wake-up method and related equipment

Abstract

The embodiment of the present application provides a voice wake-up method and related equipment. The method includes: at a first moment, the electronic device receives a first voice, and the electronic device wakes up the voice assistant; at a second moment, the electronic device receives a second voice, and the electronic device wakes up the voice assistant; the keywords in the second voice are different from the keywords in the first voice; at a third moment, the electronic device receives a third voice, and the electronic device does not wake up the voice assistant; the keywords in the third voice are different from the keywords in the first voice, and the keywords in the third voice are different from the keywords in the second voice. In this way, when the keyword is a command word, the electronic device wakes up the voice assistant; when the keyword is not a command word, the electronic device does not wake up the voice assistant; the wake-up scenarios of the voice interaction function of the electronic device can be enriched, and at the same time, the probability of the voice assistant being woken up by mistake can be reduced.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a voice wake-up method and related equipment. Background Art

[0002] Electronic devices can support voice interaction functions, and users can control smart electronic devices through voice. For example, when a user issues a voice command of "play music", a mobile phone can play music. Generally, electronic devices can include a voice assistant module. When the voice assistant module is in an awake state, the electronic device can respond to the user's voice command; when the voice assistant module is in a dormant state, the electronic device cannot respond to the user's voice command. Therefore, for electronic devices in a dormant state, the electronic device needs to be awakened first.

[0003] In a possible implementation, the user can wake up the electronic device by using a wake-up word, which can be a specific word, such as a nickname of a product. However, the user needs to say the wake-up word before using voice every time, and using the wake-up word to wake up the electronic device is not very convenient. Summary of the invention

[0004] The embodiments of the present application provide a voice wake-up method and related devices, which are applied to the field of terminal technology and can provide users with a method for waking up a voice assistant without a wake-up word, enrich the scenarios for waking up the voice assistant, simplify user operations, and enhance the user's voice interaction experience.

[0005] In the first aspect, an embodiment of the present application proposes a voice wake-up method. The method includes: at a first moment, the electronic device receives a first voice, and the electronic device wakes up the voice assistant; at a second moment, the electronic device receives a second voice, and the electronic device wakes up the voice assistant; the keywords in the second voice are different from the keywords in the first voice; the first moment is earlier than the second moment; at a third moment, the electronic device receives a third voice, and the electronic device does not wake up the voice assistant; the keywords in the third voice are different from the keywords in the first voice, and the keywords in the third voice are different from the keywords in the second voice; the second moment is earlier than the third moment. In this way, the wake-up scenarios of the voice interaction function of the electronic device can be enriched, the user operation can be simplified; and the probability of the voice assistant being woken up by mistake can be reduced.

[0006] In a possible implementation, the keywords in the first voice satisfy any one or more of the following part-of-speech combinations: verb and noun combination, noun and adjective combination, or noun and question word combination; the keywords in the second voice satisfy any one or more of the following part-of-speech combinations: verb and noun combination, noun and adjective combination, or noun and question word combination; the keywords in the third voice do not satisfy any one or more of the following part-of-speech combinations: verb and noun combination, noun and adjective combination, or noun and question word combination. In this way, the electronic device can determine whether the voice includes a command word based on whether the voice includes a fixed part-of-speech combination, which can effectively reduce the scenario of the voice assistant being mistakenly awakened and improve the accuracy of voice awakening.

[0007] In a possible implementation, the electronic device receives a first voice, and the electronic device wakes up the voice assistant, including: the electronic device obtains the first voice; the electronic device converts the first voice into a first text; the electronic device extracts text features of the first text; the semantic understanding model of the electronic device determines that the keywords in the first voice can wake up the voice assistant based on the text features; wherein the text features include the number of words in the first text, the content of the first text, and / or the word vector of the first text; the electronic device wakes up the voice assistant. In this way, the electronic device can also determine whether the voice includes a command word based on the semantic understanding model, which can effectively reduce the scenario of the voice assistant being mistakenly awakened and improve the accuracy of voice awakening.

[0008] In a possible implementation, at a first moment, the distance between the sound source of the first voice and the electronic device is the first distance; at a second moment, the distance between the sound source of the second voice and the electronic device is the first distance; at a third moment, the distance between the sound source of the third voice and the electronic device is the first distance; the method further includes: at a fourth moment, the electronic device receives the second voice, and the electronic device does not wake up the voice assistant; the third moment is later than the fourth moment; at the fourth moment, the distance between the sound source of the second voice and the electronic device is the second distance, and the second distance is greater than the first distance. In this way, when the distance between the sound source and the electronic device is far, the electronic device may not wake up the voice assistant, thereby effectively reducing the scenarios where the voice assistant is mistakenly awakened and improving the accuracy of voice awakening.

[0009] In one possible implementation, after the electronic device receives the first voice, it also includes: when the audio feature of the first voice meets the first preset condition, the electronic device calculates the sound source angle of the sound source of the first voice; wherein the audio feature includes phase and / or audio energy; the sound source angle is: the angle between the line segment from the sound source of the first voice to the center point of the electronic device and the line segment from the center point of the electronic device to the first microphone; the first microphone is the microphone on the side of the electronic device away from the front camera; when the sound source angle is less than or equal to the preset angle, the electronic device executes the process of determining whether the keywords in the first voice can wake up the voice assistant. In this way, the electronic device can determine the position of the sound source relative to the top microphone or the bottom microphone according to the sound source angle, thereby reducing the scenario where the voice assistant is mistakenly woken up due to the sound source being close to the top microphone.

[0010] In a possible implementation, the electronic device further includes a second microphone, where the second microphone is a microphone on a side of the electronic device close to the front camera; the sound source angle θ satisfies the following formula:

[0011] θ=acosd(τ au ×c / d)

[0012] Among them, τ au is the difference between the time when the first microphone obtains the first speech and the time when the second microphone obtains the first speech, c is the speed of sound, d is the distance from the first microphone to the second microphone, and acosd is the arc cosine sign. In this way, the electronic device can calculate the sound source angle according to the sound source angle formula, so that the electronic device can determine the position of the sound source.

[0013] In a possible implementation, the first preset condition includes one or more of the following: the phase difference between the first phase and the second phase is greater than a phase threshold; wherein the first phase is the phase of the first voice obtained by the first microphone, and the second phase is the phase of the first voice obtained by the second microphone; and / or, the energy difference between the first audio energy and the second audio energy is greater than an energy threshold; wherein the first audio energy is the audio energy of the first voice obtained by the first microphone, and the second audio energy is the audio energy of the first voice obtained by the second microphone. In this way, the electronic device can determine whether the first voice information is a close-talking voice based on the first preset condition, which can effectively reduce the scenario where the voice assistant is mistakenly awakened by a far-talking voice.

[0014] In a possible implementation, before the electronic device receives the first voice, it includes: when the confidence meets the second preset condition, the electronic device turns on the microphone and the voice enhancement algorithm; wherein the confidence is used to reflect the probability of the user interacting with the electronic device by voice, and the confidence is related to the inertial measurement unit IMU data of the electronic device, the image collected by the camera and / or the distance measured by the ultrasonic module; the microphone of the electronic device collects the first original voice; the electronic device receives the first voice, including: the electronic device processes the first original voice based on the voice enhancement algorithm to obtain the first voice. In this way, the electronic device can determine whether the user has the intention of voice interaction by the confidence, thereby improving the accuracy of waking up the voice assistant.

[0015] In a possible implementation, the confidence level satisfies a second preset condition, including: the first confidence level is greater than a first threshold of the first confidence level; and / or the second confidence level is greater than a first threshold of the second confidence level; and / or the third confidence level is greater than a first threshold of the third confidence level; and / or the first confidence level is greater than a second threshold of the first confidence level, and the second confidence level is greater than a second threshold of the second confidence level; and / or the first confidence level is greater than a second threshold of the first confidence level, and the third confidence level is greater than a second threshold of the third confidence level; and / or the second confidence level is greater than a second threshold of the second confidence level, and the third confidence level is greater than a second threshold of the third confidence level; wherein the first confidence level is obtained by the electronic device based on IMU data; the second confidence level is obtained by the electronic device based on a proportion of a face in an image; the third confidence level is obtained by the electronic device based on a distance between a user and an ultrasonic module; the first threshold of the first confidence level is greater than the second threshold of the first confidence level; the first threshold of the second confidence level is greater than the second threshold of the second confidence level; the first threshold of the third confidence level is greater than the second threshold of the third confidence level. In this way, the electronic device can determine whether the user has the intention of voice interaction through confidence, thereby improving the accuracy of waking up the voice assistant.

[0016] In a second aspect, an embodiment of the present application provides a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc.

[0017] The terminal device comprises: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method of the first aspect.

[0018] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method of the first aspect is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the method of the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method described in the first aspect.

[0021] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;

[0023] Figure 2A schematic diagram of the software architecture of the electronic device 100 provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a scenario of a voice wake-up method provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a process for detecting a user's voice interaction intention provided in an embodiment of the present application;

[0026] Figure 5 A flowchart of a voice wake-up method provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of a sound source angle provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of a sound source angle provided in an embodiment of the present application;

[0029] Figure 8 A flowchart of a voice wake-up method provided in an embodiment of the present application;

[0030] Fig. 9 A flowchart of a voice wake-up method provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of the structure of a voice wake-up device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0033] 1) Wake-up word: used to wake up the voice assistant to achieve voice interaction. The wake-up word can be a fixed word pre-registered in the electronic device, such as a product nickname. The voice assistant of the electronic device supports the voice interaction function; to reduce power consumption, the voice assistant remains in a dormant state in non-voice interaction scenarios; when the electronic device uses the voice interaction function, it needs to first wake up the electronic device through the wake-up word.

[0034] 2) Command words: words used to instruct the electronic device to perform corresponding functions after the voice assistant is awakened. For example, if the command word is "play music", the electronic device can run the music application in the foreground and play music after receiving the command word.

[0035] 3) Fbank: Also known as FilterBank, the human ear responds nonlinearly to the sound spectrum. Fbank is a front-end processing algorithm that processes audio in a similar way to the human ear, which can improve the performance of speech recognition. The general steps to obtain the fbank features of speech signals are: pre-emphasis, framing, windowing, short-time Fourier transform (STFT), filtering, and de-averaging.

[0036] 4) Generalized cross correlation-chase transform (GCC-PHAT): GCC-PHAT can be used to determine the sound source localization of a microphone array.

[0037] 5) Others

[0038] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0039] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0040] It should be noted that the "at..." in the embodiment of the present application can be the instant when a certain situation occurs, or a period of time after a certain situation occurs, and the embodiment of the present application does not specifically limit this. In addition, the display interface provided in the embodiment of the present application is only an example, and the display interface can also include more or less content.

[0041] In a possible implementation, the user can wake up the electronic device through a wake-up word, which can be a specific word, such as a nickname of a product. However, the user needs to say the wake-up word before using the voice interaction function of the electronic device every time, and using the wake-up word to wake up the electronic device is not very convenient. Taking the electronic device as a mobile phone as an example, every time the user interacts with the mobile phone by voice, he needs to say a wake-up word such as "Hello, XX", and the user experience is poor.

[0042] In view of this, an embodiment of the present application provides a voice wake-up method, when the user's mouth is close to the electronic device and speaks, the electronic device can collect the user's voice signal; then the electronic device can analyze whether the voice signal is a command word through parameters such as part of speech, text length corresponding to the voice signal and / or word vector; if the voice signal is a command word, the electronic device wakes up the voice assistant and responds to the command word; if the voice signal is not a command word, the electronic device does not wake up the voice assistant. In this way, when the voice assistant is dormant, the electronic device can determine whether to wake up the voice assistant through the voice signal. For example, when the user puts the mobile phone close to his mouth and says "play music", the electronic device wakes up the voice assistant and starts the music application; for another example, when the user is talking to others, the voice signal is not a command word, and the electronic device will not wake up the voice assistant; thereby expanding the scenarios in which the electronic device wakes up the voice assistant, effectively reducing the scenarios of false wake-up.

[0043] The voice wake-up method in the embodiment of the present application can be applied to electronic devices, which can also be called terminal devices, terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The terminal device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc.

[0044] Among them, wearable devices can also be called wearable smart devices, which are a general term for the intelligent design and development of wearable devices for daily wear using wearable technology, such as glasses, gloves, watches, clothing and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also realize powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and independent of smartphones to achieve complete or partial functions, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0045] In addition, in the embodiments of the present application, the terminal device may also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0046] In an embodiment of the present application, the terminal device may include a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system may be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0047] In order to better understand the embodiments of the present application, the structure of the electronic device in the embodiments of the present application is introduced below:

[0048] Figure 1The structure diagram of the electronic device 100 is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor 180B, an air pressure sensor, a magnetic sensor, an acceleration sensor 180E, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0049] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0050] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0051] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0052] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0053] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0054] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0055] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0056] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0057] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0058] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0059] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0060] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0061] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0062] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0063] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0064] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0065] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0066] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0067] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0068] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0069] The receiver 170B, also called a "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 receives a call or a voice signal, the voice can be received by placing the receiver 170B close to the human ear.

[0070] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice signal, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0071] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0072] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0073] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0074] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0075] Figure 2 1 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.

[0076] The layered architecture of the electronic device 100 divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system may include an application layer, an application framework layer, a system library, and a kernel layer.

[0077] The application layer can include a series of application packages.

[0078] like Figure 2 As shown, the application package may include applications such as camera, calendar, map, WLAN, Bluetooth, short message, gallery, call, video, voice assistant, etc.

[0079] Among them, the voice assistant is an intelligent application that helps users solve problems through intelligent interaction of intelligent dialogue and instant question and answer. The voice assistant application is also referred to as the voice assistant module in this article.

[0080] The semantic understanding module is also called a natural language understanding (NLU) module. In the embodiment of the present application, the semantic understanding module can be used to obtain the semantic representation of the speech signal through grammar, semantics, and pragmatics analysis.

[0081] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0082] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a resource manager, a view system, a notification manager, an activity manager, and the like.

[0083] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, capture the screen, etc.

[0084] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0085] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0086] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0087] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0088] The activity manager is responsible for managing the startup, status, life cycle and other activities of the application. Among them, activity is an application component that can provide an interface for users to interact with the terminal device through the interface to complete a task.

[0089] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0090] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0091] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0092] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0093] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0094] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0095] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0096] A 2D graphics engine is a drawing engine for 2D drawings.

[0097] The kernel layer is the layer between hardware and software. The kernel layer can contain display drivers, camera drivers, audio drivers, sensor drivers, etc.

[0098] ADSP (audio digital signal processor) is connected to the kernel layer. ADSP is used to process audio data, such as voice data. When the main processor in the electronic device is dormant, ADSP can process audio data and sensor data to reduce power consumption.

[0099] In the embodiment of the present application, the ADSP may include a wake-up detection module, a speech enhancement algorithm module, and a wake-up judgment module. Among them, the wake-up detection module can be used to predict the probability of a user performing speech interaction with an electronic device, and the wake-up detection module can identify whether the user has a speech interaction intention; the speech enhancement algorithm module can be used to perform audio processing such as echo cancellation, noise reduction, and reverberation on the speech signal; the wake-up judgment module can be used to determine whether the speech signal is a close-talking speech, and whether the sound source is close to the bottom microphone.

[0100] Understandably, Figure 2 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0101] In the embodiment of the present application, the electronic device can wake up the voice assistant in the scenario where the user does not use the wake-up word, so as to realize the voice interaction between the electronic device and the user. The following takes the electronic device as a mobile phone as an example to illustrate a voice wake-up method provided in the embodiment of the present application. The electronic device can also be other product forms, for example, the electronic device can also be a computer, a tablet and a wearable device, etc., and the embodiment of the present application does not limit this.

[0102] Figure 3 A scenario diagram of a voice wake-up method in an embodiment of the present application is shown, such as Figure 3 As shown:

[0103] For example, Figure 3 In the scenario shown in a, the user may be using an electronic device; for example, the user is viewing a photo on an electronic device, and the electronic device may display the following: Figure 3 The interface shown in c in the figure, where Figure 3The interface shown in c includes photos.

[0104] In some scenarios, when the electronic device does not start the voice assistant, the user may have the intention of voice interaction, for example, the user wants to call friend A. At this time, the user can bring the electronic device close to the mouth and say "call A", such as Figure 3 In response to the user's voice, the electronic device may display the following Figure 3 The interface shown in d; Figure 3 In the interface shown in d, the electronic device initiates a call to the electronic device of friend A.

[0105] It can be seen from the above scenario that during the voice interaction between the user and the electronic device, the electronic device did not receive the wake-up word; instead, after receiving the user's voice "Call A", the electronic device responded to the voice and initiated a call to the electronic device held by friend A.

[0106] This is because, in the embodiment of the present application, the electronic device can preliminarily determine whether the user has the intention of voice interaction based on conditions such as user operation and / or the distance between the electronic device and the user, and then the electronic device can determine whether to execute the instruction corresponding to the voice signal based on the distance between the source and the electronic device, the text features of the voice signal, etc. For example, whether the user has the intention of voice interaction can be reflected by the following factors: whether the user performs a lifting action, whether the user faces or looks at the electronic device, and / or the distance between the user and the electronic device. In this way, the electronic device can wake up the voice assistant and execute the instruction corresponding to the voice signal without receiving the wake-up word.

[0107] Combine the following Figure 4-Figure 8 A voice wake-up method provided in an embodiment of the present application is described in detail.

[0108] In order to reduce the power consumption of electronic devices, the embodiments of the present application can first determine whether the user has the intention of voice interaction. When the user has the intention of voice interaction, the electronic device turns on the microphone and the voice enhancement algorithm, thereby avoiding the microphone and the voice enhancement algorithm being turned on for a long time.

[0109] It should be noted that a first flag is set in the embodiment of the present application, and the value of the first flag includes a first value and a second value; wherein the first flag can reflect whether the user has a voice interaction intention; for example, when the first flag is the first value, the user may have a voice interaction intention, and the electronic device can turn on the microphone and the voice enhancement algorithm to facilitate the subsequent collection and processing of the voice signal emitted by the user. For another example, when the first flag is the second value, the user may not have a voice interaction intention, the electronic device does not continue to execute the voice wake-up method, and the electronic device does not turn on the microphone and the voice enhancement algorithm.

[0110] Exemplarily, the present application embodiment provides four methods for an electronic device to determine whether a user has a voice interaction intention, and the four methods are as follows:

[0111] The first method: the electronic device identifies whether the user operation is a lifting operation based on inertial measurement unit (IMU) data; when the user operation is a lifting operation, the electronic device may set the value of the first flag bit to the first value.

[0112] When the user performs the lifting operation, the user may have the intention of voice interaction, and the electronic device can continue to perform subsequent steps through the first flag bit of the first value, for example, turning on the microphone to collect voice signals, turning on the voice enhancement algorithm to process the voice signals to obtain the distance from the sound source to the electronic device, etc. When the user does not perform the lifting operation, the user may not have the intention of voice interaction, and the electronic device can terminate the subsequent process through the first flag bit of the second value and re-monitor the IMU data.

[0113] It is understandable that in a conventional scenario, when a user wants to interact with a mobile phone by voice, the user often picks up the mobile phone and brings it close to his face. During this process, the user lifts the mobile phone. Therefore, in the embodiment of the present application, whether the electronic device recognizes the user's lifting operation can be used as a judgment condition for the user's desire to interact by voice.

[0114] IMU data may be data collected by sensors of electronic devices. For example, IMU data may include angular velocity and acceleration of the electronic device around three axes (i.e., x-axis, y-axis, and z-axis) collected by a gyroscope sensor or acceleration of the electronic device in various directions (generally three axes) collected by a linear acceleration sensor.

[0115] Specifically, three methods for an electronic device to recognize a lifting operation are provided below.

[0116] In one possible implementation, the electronic device may extract data features of the IMU data, and use neural networks (NN) and the data features of the IMU data to determine whether the user's current action is a lifting operation.

[0117] For example, the data features of IMU data can be the variance and / or average of the length of movement of the electronic device in the three-axis direction, the rotation angle of the electronic device, and / or the frequency of change of the three-axis data, etc.; the three-axis data can include the data of the x-axis, y-axis and z-axis collected by the gyroscope sensor, and / or the data of the x-axis, y-axis and z-axis collected by the acceleration or linear acceleration sensor. In some embodiments, the NN network can be used for posture solution, and the electronic device can score the IMU data through the NN network. When the score is higher than the score threshold, the electronic device determines that the user performs a lifting operation.

[0118] It should be noted that different NN networks correspond to different score thresholds, and this application does not impose any restrictions on this.

[0119] In another possible implementation, the electronic device may use a low-pass filter to filter the original IMU data to obtain filtered IMU data; the electronic device obtains changes in numerical values ​​of multiple axes within a preset time period based on the filtered IMU data; the electronic device determines whether the user's current action is a lifting operation through changes in numerical values ​​of multiple axes.

[0120] For example, the change of the values ​​of the multiple axes may be the rotation angle of the electronic device and / or the fluctuation range of the electronic device, and the preset time length may be 1 second, 2 seconds or other time lengths. When the change of the values ​​of the multiple axes meets the change threshold of the respective axes, the electronic device determines that the user performs a lifting operation.

[0121] The above method can obtain the judgment result of the user action based on the calculation, and the electronic device occupies less computing power during the calculation process.

[0122] In another possible implementation, the electronic device may combine the above two methods to obtain the determination result of the user action. For example, the electronic device may first score the IMU data through the NN network, and then the electronic device verifies the scoring result by calculating the IMU data.

[0123] When the electronic device uses the above two methods at the same time, the electronic device can obtain the determination result of the user action more accurately, thereby improving the accuracy of the voice wake-up method of the embodiment of the present application.

[0124] The second method: the camera of the electronic device captures an image; the electronic device obtains the proportion of the face in the image based on the face recognition module; when the proportion of the face in the image shows an increasing trend, the electronic device can set the value of the first flag bit to the first value.

[0125] The increasing proportion of faces in images can be understood as: when the user lifts the electronic device, the camera of the electronic device gets closer and closer to the user's face, causing the face in the image captured by the camera to increase from small to large, or from the image captured by the camera that does not include the face to the image that includes the face; the camera may be the front camera of the electronic device.

[0126] In some embodiments, the electronic device may obtain that the proportion of human faces in the image is increasing within a preset time. Exemplarily, when the duration of the increasing proportion of human faces in the image is less than the preset time, the electronic device may set the value of the first flag to the first value; when the duration of the increasing proportion of human faces in the image is greater than or equal to the preset time, the electronic device may set the value of the first flag to the second value.

[0127] It is understandable that the user's action of lifting the electronic device is a relatively short process. Therefore, the duration for the electronic device to detect that the proportion of the face in the image is increasing should be within a preset time, for example, the preset time can be 5 seconds. If the preset time is exceeded, the electronic device can determine that the action performed by the user is not a lifting operation. In this way, the voice assistant can be mistakenly awakened in some scenarios, and the accuracy of the voice awakening method can be improved.

[0128] A third method: the electronic device measures the distance between the user's face and the electronic device based on the ultrasonic signal; when the distance is less than the distance threshold, the electronic device may set the value of the first flag bit to the first value.

[0129] In some embodiments, an ultrasonic module is provided in the electronic device, and the electronic device can transmit an ultrasonic signal through the ultrasonic module, and calculate the distance between the electronic device and the user's face according to the returned ultrasonic signal.

[0130] It is understandable that when a user wants to interact with an electronic device by voice, the user often places the electronic device near his mouth and speaks a command word to the electronic device. In this scenario, the distance between the electronic device and the user's face may be less than the distance threshold. Therefore, when the distance between the electronic device and the user's face is less than the distance threshold, the electronic device may continue to execute the subsequent steps in the voice wake-up method; when the distance between the electronic device and the user's face is greater than or equal to the distance threshold, the electronic device may terminate the voice wake-up method.

[0131] In an embodiment of the present application, the electronic device may include multiple microphones. For example, the microphones may include: a top microphone, a bottom microphone, and / or a back microphone. The top microphone may be a microphone on the side of the electronic device close to the front camera. The bottom microphone may be a microphone on the side of the electronic device far from the front camera. The back microphone may be a microphone on the side of the electronic device close to the rear camera. In some embodiments, the ultrasonic module of the electronic device may be arranged close to the bottom microphone. The distance obtained by the electronic device based on the ultrasonic signal may also be understood as the distance between the user's face and the bottom microphone.

[0132] It should be noted that in the embodiments of the present application, the electronic device may use the first method, the second method or the third method to determine whether the user has the intention of voice interaction.

[0133] Taking the example of an electronic device using the first method to determine whether a user has the intention of voice interaction, for example, the electronic device can obtain a first confidence level based on IMU data. When the first confidence level is greater than the threshold of the first confidence level, the electronic device can set the first flag bit to the first value.

[0134] Taking the example of an electronic device using the second method to determine whether a user has the intention of voice interaction, for example, the electronic device can obtain a second confidence level based on the proportion of the face in the image. When the second confidence level is greater than the threshold of the second confidence level, the electronic device can set the first flag bit to the first value.

[0135] Taking the example of an electronic device using the third method to determine whether a user has the intention of voice interaction, for example, the electronic device can obtain a third confidence level based on the distance between the user and the ultrasonic module. When the third confidence level is greater than the threshold of the third confidence level, the electronic device can set the first flag bit to the first value.

[0136] The fourth method: The electronic device can use a fusion of the above three methods to determine whether the user has the intention of voice interaction. Exemplarily, the electronic device can combine the first method with the second method, the first method with the third method, or the second method with the third method; the electronic device can also combine the first method, the second method and the third method. It can be understood that when the electronic device determines the user's voice interaction intention from multimodal information, the determination result obtained by the electronic device is more accurate. The embodiment of the present application does not limit the combination method.

[0137] The following takes the first method, the second method and the third method as examples. Figure 4 The fusion process is described.

[0138] Exemplarily, S401, the electronic device outputs a first confidence level according to IMU data.

[0139] The first confidence level can be used to reflect the probability that the electronic device obtains the user's voice interaction intention based on the IMU data. For example, the first confidence level can be a score output by the NN network and / or a calculation result output based on the IMU data.

[0140] S402: The electronic device outputs a second confidence level according to an image captured by a camera.

[0141] The second confidence level may be used to reflect the probability that the electronic device obtains the user's voice interaction intention based on the image captured by the camera. For example, the second confidence level may be the proportion of the face in the image.

[0142] S403: The electronic device outputs a third confidence level according to the ultrasonic signal.

[0143] The third confidence level can be used to reflect the probability that the electronic device obtains the user's voice interaction intention based on the ultrasonic signal. The third confidence level can be the distance between the user and the ultrasonic module.

[0144] S404: When the confidence level meets the second preset condition, the electronic device turns on the microphone and the speech enhancement algorithm.

[0145] The confidence level is used to reflect the probability of a user performing voice interaction with an electronic device. The confidence level is related to the inertial measurement unit (IMU) data of the electronic device, the image captured by the camera, and / or the distance measured by the ultrasonic module.

[0146] Exemplarily, the confidence level satisfies the second preset condition, including:

[0147] a) The first confidence level is greater than a first threshold of the first confidence level.

[0148] b) and / or, the second confidence level is greater than a first threshold value of the second confidence level.

[0149] c) and / or, the third confidence level is greater than a first threshold value of the third confidence level.

[0150] d) and / or, the first confidence is greater than a second threshold of the first confidence, and the second confidence is greater than a second threshold of the second confidence.

[0151] e) and / or, the first confidence is greater than a second threshold of the first confidence, and the third confidence is greater than the second threshold of the third confidence.

[0152] f) and / or, the second confidence is greater than a second threshold value of the second confidence, and the third confidence is greater than the second threshold value of the third confidence.

[0153] In the embodiment of the present application, the threshold of any confidence level may include a first threshold and a second threshold, wherein the first threshold is greater than the second threshold. For ease of explanation, the first threshold may be understood as a high threshold in the threshold range corresponding to any confidence level, and the second threshold may be understood as a low threshold in the threshold range corresponding to any confidence level.

[0154] For example, for ease of description, the first threshold of the first confidence level may be the third threshold level, and the second threshold of the first confidence level may be the sixth threshold level; the third threshold level is greater than the sixth threshold level. If the first confidence level is greater than the third threshold level, it can be understood that the probability that the user has a voice interaction intention is high; if the first confidence level is less than the sixth threshold level, it can be understood that the probability that the user has a voice interaction intention is low.

[0155] For another example, for ease of description, the first threshold of the second confidence level may be the fourth threshold, and the second threshold of the second confidence level may be the seventh threshold; the fourth threshold is higher than the seventh threshold. If the second confidence level is greater than the fourth threshold, it can be understood that the probability that the user has a voice interaction intention is high; if the second confidence level is less than the seventh threshold, it can be understood that the probability that the user has a voice interaction intention is low.

[0156] For another example, for ease of description, the first threshold of the third confidence level may be the fifth threshold, the second threshold of the third confidence level may be the eighth threshold, and the fifth threshold is greater than the eighth threshold. If the third confidence level is greater than the fifth threshold, it can be understood that the probability that the user has a voice interaction intention is high; if the third confidence level is less than the eighth threshold, it can be understood that the probability that the user has a voice interaction intention is low.

[0157] Based on the above confidence threshold, when the confidence meets the second preset condition, the electronic device turns on the microphone and the voice enhancement algorithm, which may correspond to the following two scenarios, for example:

[0158] (1) When any confidence level is greater than a first threshold, the electronic device turns on the microphone and the speech enhancement algorithm.

[0159] For example, the confidence level satisfies the conditions a, b and c in the second preset condition. Specifically, the first confidence level is greater than the third threshold; and / or the second confidence level is greater than the fourth threshold; and / or the third confidence level is greater than the fifth threshold.

[0160] It can be understood that a confidence level higher than the high threshold among the confidence thresholds may indicate that the probability that the user has the intention of voice interaction is very high. Therefore, when any one of the first to third confidence levels is higher than the high threshold corresponding to the confidence level, it can be considered that the user has the intention of voice interaction, and the electronic device can set the value of the first flag bit to the first value.

[0161] (2) When any two confidence levels are greater than a second threshold, the electronic device turns on the microphone and the speech enhancement algorithm.

[0162] For example, the confidence level satisfies the conditions d, e, and f in the second preset condition.

[0163] Specifically, condition d may correspond to: the first confidence is greater than the sixth threshold, the first confidence is less than or equal to the third threshold; and the second confidence is greater than the seventh threshold, the second confidence is less than or equal to the fourth threshold; and the third confidence is less than or equal to the eighth threshold.

[0164] The e condition may correspond to: the first confidence is greater than the sixth threshold, the first confidence is less than or equal to the third threshold; and the third confidence is greater than the eighth threshold, the third confidence is less than or equal to the fifth threshold; and the second confidence is less than or equal to the seventh threshold.

[0165] The f condition may correspond to: the second confidence is greater than the seventh threshold, the second confidence is less than or equal to the fourth threshold; and the third confidence is greater than the eighth threshold, the third confidence is less than or equal to the fifth threshold; and the first confidence is less than or equal to the sixth threshold.

[0166] Condition d, condition e and condition f may also correspond to: the first confidence is greater than the sixth threshold, and the first confidence is less than or equal to the third threshold; and the second confidence is greater than the seventh threshold, and the second confidence is less than or equal to the fourth threshold; and the third confidence is greater than the eighth threshold, and the third confidence is less than or equal to the fifth threshold.

[0167] It is understandable that when only one confidence level is between the high threshold and the low threshold, and the other confidence levels are less than or equal to the low threshold, the electronic device may not be able to accurately determine whether the user has the intention of voice interaction; but when multiple confidence levels are between the high threshold and the low threshold, the accuracy of the electronic device's determination results can be improved, so that the lower confidence level can still be used as a basis for determining whether the user has the intention of voice interaction, rather than invalid data.

[0168] It should be noted that an embodiment of the present application provides a method for an electronic device to determine whether a user has a voice interaction intention based on three confidence levels. In actual scenarios, the electronic device may also determine whether a user has a voice interaction intention based on one or two confidence levels, and the embodiment of the present application does not limit this. In addition, the embodiment of the present application obtains the first to third confidence levels in the order of steps S401-S403; the present application may also obtain the first to third confidence levels in other orders, for example, the execution order may also be steps S401, S403, and S402, and the embodiment of the present application does not limit this.

[0169] The electronic device may perform step S401 based on the sensor, perform step S402 based on the camera, and perform step S403 based on the ultrasonic module, so that the sensor, camera, ultrasonic module, etc. are in a normally open state, and the power consumption of the electronic device is relatively large. In some embodiments, the electronic device may first determine the distance between the electronic device and the user based on the sensor and ultrasonic module, and when the distance is less than the distance threshold, the electronic device may turn on the camera to collect images. In this way, the power consumption of some components can be reduced.

[0170] It should be noted that for devices that need to be awakened by a wake-up word, the microphone and speech enhancement algorithm in the electronic device are in a normally-on state. For example, the microphone needs to remain on to ensure that the electronic device can receive the wake-up word and command word spoken by the user in a timely manner. In the embodiment of the present application, the electronic device does not need to set the microphone and speech enhancement algorithm to a normally-on state. The electronic device can turn on the microphone and speech enhancement algorithm when the first flag bit is the first value, thereby reducing the power consumption of the microphone and speech enhancement algorithm.

[0171] In an embodiment of the present application, the electronic device outputs a first confidence level based on the IMU data; the electronic device outputs a second confidence level based on the image captured by the camera; the electronic device outputs a third confidence level based on the ultrasonic signal; when the confidence level meets the second preset condition, the electronic device turns on the microphone and the speech enhancement algorithm; in this way, the judgment result obtained by the electronic device based on the multi-mode information is more accurate, which can reduce the probability of false wake-up and improve the accuracy of voice wake-up; at the same time, it can also avoid the voice enhancement algorithm and the wake-up judgment algorithm from being always on, thereby reducing power consumption.

[0172] The above embodiment describes the method for determining the user's voice interaction intention. Figure 5 The subsequent process is described. Figure 5 A voice wake-up method provided by an embodiment of the present application is shown, such as Figure 5 As shown:

[0173] Exemplarily, S501, when the first flag bit is a first value, the electronic device turns on a voice enhancement algorithm and a microphone.

[0174] The speech enhancement algorithm is used to process the speech signal, for example, echo cancellation, noise reduction and reverberation removal, etc. The microphone includes a top microphone, a bottom microphone and a back microphone.

[0175] It is understandable that when the first flag bit is the first value, the electronic device can determine that the user has a voice interaction intention, and at this time, the electronic device can turn on the microphone and the voice enhancement algorithm to enable the electronic device to collect the user's voice signal and process the voice signal. In this way, the voice enhancement algorithm and the microphone can be prevented from being turned on for a long time, thereby reducing the power consumption of the electronic device.

[0176] S502: The microphone of the electronic device collects an original voice signal, and the electronic device processes the original voice signal based on a voice enhancement algorithm to obtain a voice signal.

[0177] The original voice signal may include original voice signals collected by multiple microphones. For example, the original voice signal may include: an original voice signal collected by a bottom microphone, an original voice signal collected by a top microphone, and an original voice signal collected by a back microphone.

[0178] The speech enhancement algorithm may include an echo cancellation algorithm, a denoising algorithm, and a dereverberation algorithm, etc. The electronic device may process the original speech signal of the bottom microphone using the echo cancellation algorithm, the denoising algorithm, and the dereverberation algorithm to obtain the speech signal of the bottom microphone; the electronic device may process the original speech signal of the top microphone using the echo cancellation algorithm, the denoising algorithm, and the dereverberation algorithm to obtain the speech signal of the top microphone; the electronic device may process the original speech signal of the back microphone using the echo cancellation algorithm, the denoising algorithm, and the dereverberation algorithm to obtain the speech signal of the back microphone.

[0179] S503: The electronic device determines whether the audio feature of the speech signal meets a first preset condition based on the audio feature of the speech signal.

[0180] The audio features of speech signals include: Fbank features, audio energy and phase, etc.

[0181] It is understandable that when the distance from the sound source to the microphone is small, the user's mouth is close to the electronic device, and the user's voice signal may be a close-talking voice. When the distance from the sound source to the microphone is large, the user's mouth is far away from the electronic device, and the user's voice signal may be a far-talking voice. Based on conventional understanding, the scenario in which the user's mouth is far away from the electronic device is inconsistent with the scenario in which the user interacts with the electronic device by voice; therefore, in an embodiment of the present application, whether the scene in which the user speaks conforms to the voice interaction scenario can be reflected by whether the voice signal is a close-talking voice. If the user's voice signal is a close-talking voice, the electronic device may continue to perform subsequent operations to wake up the voice assistant; if the user's voice signal is not a close-talking voice, the electronic device may not perform the operation of waking up the voice assistant.

[0182] In an embodiment of the present application, the first preset condition is used to reflect whether the user's voice signal is a close-talking voice. The first preset condition is related to the audio characteristics of the voice signal. In an embodiment of the present application, the wake-up judgment model of the electronic device can output whether the user's voice signal is a close-talking voice based on the audio characteristics of the voice signal. Among them, the wake-up judgment model can be set in a wake-up judgment module of the electronic device, and the wake-up judgment model may include: a convolutional neural network-gated recurrent unit (CNN-GRU).

[0183] In some embodiments, the electronic device may determine whether the user's voice signal is a close-talking voice based on the audio energy difference between the voice signals.

[0184] For example, the audio energy may be an acoustic signal level. When a user speaks close to an electronic device, the audio energy of the voice signals obtained by the microphones in the electronic device may differ. For example, the acoustic signal level may decrease by 6 dB for every doubling of the distance between the microphone and the sound source. The electronic device may determine whether the voice signal is a close-talking voice based on the audio energy difference between the voice signals.

[0185] For example, taking the speech signal of the bottom microphone and the speech signal of the top microphone as an example, the first preset condition includes: the energy difference between the first audio energy and the second audio energy is greater than the energy threshold; wherein the first audio energy is the audio energy of the speech signal obtained by the bottom microphone, and the second audio energy is the audio energy of the speech signal obtained by the top microphone.

[0186] Among them, when the energy difference between the speech signal of the bottom microphone and the speech signal of the top microphone is greater than the energy threshold, the speech signal can be a near-speaking speech; when the audio energy difference between the speech signal of the bottom microphone and the speech signal of the top microphone is less than or equal to the audio energy threshold, the speech signal can be a far-speaking speech.

[0187] In other embodiments, the electronic device may determine whether the user's voice signal is a close-talking voice based on the phase difference between the voice signals.

[0188] For example, when a user speaks close to an electronic device, the phases of the voice signals obtained by the microphones in the electronic device are different. The electronic device can determine whether the voice signal is a close-talking voice based on the phase difference between the voice signals.

[0189] For example, taking the speech signal of the bottom microphone and the speech signal of the top microphone as an example, the first preset condition also includes: the phase difference between the first phase and the second phase is greater than the phase threshold; wherein the first phase is the phase of the speech signal obtained by the bottom microphone, and the second phase is the phase of the speech signal obtained by the top microphone.

[0190] Among them, when the phase difference between the voice signal of the bottom microphone and the voice signal of the top microphone is greater than the phase threshold, the voice signal can be a near-speaking voice; when the phase difference between the voice signal of the bottom microphone and the voice signal of the top microphone is less than or equal to the phase threshold, the voice signal can be a far-speaking voice.

[0191] It should be noted that in the embodiment of the present application, the wake-up judgment model can also judge whether the user's voice signal is a close-talking voice based on other audio features. The embodiment of the present application does not give examples one by one. In the embodiment of the present application, the energy difference value and the phase difference value can also be the difference between the voice signal of the bottom microphone and the voice signal of the back microphone or the difference between the voice signal of the top microphone and the voice signal of the back microphone, and the embodiment of the present application does not limit this. In addition, in the embodiment of the present application, there is no specific limitation on the energy threshold, phase threshold and the dividing line between close-talking voice and far-talking voice.

[0192] S504: When the audio feature of the speech signal meets the first preset condition, the electronic device calculates the sound source angle of the sound source of the speech signal.

[0193] In the embodiment of the present application, the sound source angle is: the angle between the line segment from the sound source to the center point of the electronic device and the line segment from the center point of the electronic device to the bottom microphone. Figure 6 As shown:

[0194] Among them, the top microphone may correspond to point A, the bottom microphone may correspond to point B, the sound source may correspond to point C, and the midpoint of the electronic device may correspond to point D; line segment AB may correspond to the distance from the top microphone to the bottom microphone in the electronic device; straight line EF may be the midline of line segment AB, and the angle between line segment CD and line segment AB is the sound source angle.

[0195] It has been found that in actual scenarios, when users interact with electronic devices by voice, they tend to put their mouths close to the bottom microphone of the electronic device. Therefore, in the embodiment of the present application, if the sound source is close to the bottom microphone, it means that the user may be interacting with the electronic device by voice; if the sound source is close to the top microphone, it means that the user may be speaking to the electronic device but is not interacting with the electronic device by voice. Among them, when the sound source angle θ is between 0°-90°, it can be considered that the sound source is close to the bottom microphone, and when the sound source angle θ is between 90°-180°, it can be considered that the sound source is close to the top microphone. The electronic device can further confirm whether the user is interacting with the electronic device based on the sound source angle.

[0196] Combine the following Figure 7 This paper introduces a method for calculating the sound source angle. Figure 7 As shown:

[0197] In an embodiment of the present application, the electronic device can use GCC-PHAT to determine the direction of the sound source. Exemplarily, the electronic device calculates the time delay between the top microphone and the bottom microphone based on the generalized cross-correlation function; and the electronic device calculates the sound source angle according to the time delay.

[0198] The delay can be the time difference between the time when the speech signal arrives at the top microphone and the time when the speech signal arrives at the bottom microphone, for example Figure 7 As shown in b, when the sound source is near the straight line EF, AC = BC, the speech signal can reach the top microphone and the bottom microphone at the same time, and the delay can be 0. When the sound source is on the right side of the straight line EF, AC>BC, as shown in Figure 7 As shown in a, the speech signal can reach the bottom microphone first and then the top microphone. The delay can be the time it takes for the speech signal to reach the top microphone minus the time it takes for the speech signal to reach the bottom microphone. The delay is a positive value. Similarly, when the sound source is on the left side of the straight line EF, the delay is a negative value.

[0199] In one possible implementation, the electronic device may determine the direction of the sound source based on the positive or negative value of the time delay; for example, if the time delay is a positive value, the sound source is close to the bottom microphone; if the time delay is a negative value, the sound source is close to the top microphone.

[0200] To improve the accuracy of the sound source orientation, in the embodiment of the present application, the electronic device can further calculate the sound source angle based on the time delay. Figure 7 As shown in a, CI = BI, where AI is the distance corresponding to the delay. The sound source angle θ can satisfy the following formula:

[0201] θ=acosd(τ au ×c / d)

[0202] Among them, τ au is the time delay, c is the speed of sound, and d is the distance from the top microphone to the bottom microphone.

[0203] Delay τ au It is obtained based on the following method, which is exemplary:

[0204] The electronic device performs windowing processing and Fourier transform processing on the speech signal of the bottom microphone and the speech signal of the top microphone respectively to obtain a first frequency domain signal and a second frequency domain signal; the electronic device performs cross-correlation analysis on the first frequency domain signal and the second frequency domain signal to obtain a first function; the electronic device performs peak extraction on the first function to obtain a time delay.

[0205] Among them, windowing processing can be used to reduce spectrum leakage, and Fourier transform is used to process the speech signal in the time domain into a speech signal in the frequency domain; the frequency domain signal may include amplitude information and phase information. The first function can be a function obtained by multiplying the frequency of the first frequency domain signal by the frequency of the second frequency domain signal. In the first function, when the delay is at the maximum value, a peak value can be obtained, and the peak value corresponds to the delay. The electronic device obtains the delay by peak extraction.

[0206] The embodiment of the present application only introduces a method for obtaining the direction of the sound source by way of example. The embodiment of the present application may also use other methods to determine the direction of the sound source. For example, the electronic device may also use the sound source angle formula to perform preliminary calculations based on steps S503 and S504, and then divide the distance between the top microphone and the bottom microphone into 180 delays on average, and represent the sound source angle by the position of the sound source at the 180 delays. The embodiment of the present application does not limit this.

[0207] S505: When the sound source angle of the sound source is less than or equal to the preset angle, the electronic device executes a process of determining whether the voice signal can wake up the voice assistant.

[0208] The preset angle may be 90°. The process of determining whether a voice signal can wake up a voice assistant may enable the electronic device to perform semantic understanding of the voice signal.

[0209] In one possible implementation, the electronic device may reflect whether the electronic device performs a semantic understanding process through a second flag. For example, the second flag is used to indicate whether the electronic device performs semantic understanding on a voice signal. The value of the second flag may include a third value and a fourth value; when the sound source angle of the sound source is less than or equal to a preset angle, the electronic device may continue to execute subsequent steps through the second flag of the third value, for example, using a semantic understanding module to determine whether the voice signal is a command word; when the sound source angle of the sound source is greater than a preset angle, the electronic device may terminate the execution of subsequent steps through the second flag of the fourth value.

[0210] In the embodiment of the present application, the electronic device can determine whether to continue the subsequent voice wake-up process based on the sound source angle; when the sound source angle of the sound source is less than or equal to the preset angle, the electronic device executes the process of determining whether the voice signal can wake up the voice assistant. In this way, the scenario where the electronic device mistakenly wakes up the voice assistant when the user speaks close to the top microphone can be reduced, thereby improving the accuracy of the voice wake-up method.

[0211] In some scenarios, the user puts the bottom microphone of the mobile phone close to the mouth and chats with friends at the same time. At this time, the chat content is not a command word, but because the sound source is near the bottom microphone, the electronic device may mistakenly wake up the voice assistant. To solve this problem, in the embodiment of the present application, the electronic device can also further determine whether the voice signal is a command word through a semantic understanding model.

[0212] For example, in the embodiment of the present application, the electronic device may convert the voice signal into text information based on voice recognition; the electronic device may determine whether the text information includes a command word based on part-of-speech tagging and / or a semantic understanding model. The voice signal may be a voice signal from a top microphone.

[0213] In a possible implementation, the electronic device may determine whether the text information includes a command word through part-of-speech tagging.

[0214] Part-of-speech tagging can be used to describe the structural rules of sentences. Parts of speech may include adjectives, nouns, verbs, and quantifiers. Electronic devices can use sentences that meet certain part-of-speech combinations as command words and respond to the command words after waking up the voice assistant.

[0215] The command word satisfies any one or more of the following part-of-speech combinations: a verb and a noun combination, a noun and an adjective combination, or a noun and an interrogative word combination. The specific part-of-speech combinations can be set according to the product or scenario and are not listed one by one.

[0216] The command word satisfies the combination of verbs and nouns. For example, the command word may be "open / mobile phone", "play / music", "tell / a / story", "set / alarm clock" and "call / friend A", etc. The embodiments of the present application will not list them one by one. It can be seen that the above command words may include verbs and nouns. The electronic device may use the sentence that satisfies the part of speech combination of verb + noun as the command word. The electronic device may wake up the voice assistant and respond to the sentence.

[0217] And / or, the command word satisfies the combination of noun and adjective, for example, the command word may be "voice / lower", "screen / brighter", etc. It can be seen that the above command words may include nouns and adjectives, and the electronic device may use the sentence that satisfies the part of speech combination of noun + adjective as the command word. The electronic device may wake up the voice assistant and respond to the sentence.

[0218] And / or, the command word satisfies nouns and question words, for example, the command word can be "how is the weather / ", "what do I have / schedule", "what day is / is today", etc. It can be seen that the above command words can include nouns and question words, and the electronic device can use the sentence that satisfies the part of speech combination of noun + question word as the command word. The electronic device can wake up the voice assistant and respond to the sentence.

[0219] In another possible implementation, the electronic device may determine whether the text information includes a command word through a semantic understanding model.

[0220] The semantic understanding model can be trained based on a corpus database, and the semantic understanding model has the ability to identify whether text information is a command word. The semantic understanding model of the electronic device can determine whether the text is a command word based on text features. The text features include the number of words in the text corresponding to the voice signal, the content of the text, and / or the word vector of the text;

[0221] In some scenarios, in order to improve the accuracy of electronic devices responding to command words, the number of words or voice length of command words is often not too long. Therefore, the semantic understanding model of the electronic device can determine whether the text includes command words based on the number of words in the recognized text information; when the number of words in the text information is less than or equal to the word count threshold, the electronic device can output the text information as a command word; when the number of words in the text information is greater than the word count threshold, the electronic device can output that the text information is not a command word.

[0222] Specifically, the electronic device can learn the semantic understanding model based on the corpus sample. For example, the corpus sample includes command words and non-command words, the command words are marked as 1, and the non-command words are marked as 0; the semantic understanding model obtains the learned semantic understanding model based on the number of words in the command words. The learned semantic understanding model has the ability to match the text of the voice signal with the command words.

[0223] In other scenarios, the semantic understanding model of the electronic device can determine whether the text information includes a command word based on the word vector.

[0224] For example, the semantic understanding model can learn the association between words based on a large amount of corpus, and the word vector can be represented by numbers. For example, taking "open the mobile phone" as an example, the semantic understanding model can learn the word vector of "open" as vector 1 and "mobile phone" as vector 2; taking "open the gallery application" as an example, the semantic understanding model can learn the word vector of "open" as vector 1 and "gallery application" as vector 3.

[0225] In the embodiment of the present application, after determining that the text information is a command word, the electronic device can transmit the text information to the voice assistant, and the voice assistant responds according to the text information. For example, the text information is "play music"; the voice assistant responds according to the text information and can open the music application for the electronic device and play music.

[0226] In the embodiment of the present application, the semantic understanding model can also determine whether the user's voice is a command word based on the content of the text; the semantic understanding model can also determine whether the user's voice is a command word by combining multiple text features. This embodiment of the present application will not be listed one by one.

[0227] In the embodiment of the present application, the electronic device can determine whether the user's voice is a command word by semantically understanding the voice signal. When the text information is a command word, the electronic device wakes up the voice assistant; when the text information is not a command word, the electronic device does not wake up the voice assistant. In this way, the scenario where the electronic device mistakenly wakes up the voice assistant when the user is talking to others can be reduced, thereby improving the accuracy of the voice wake-up method.

[0228] Combine the following Figure 8 The process of the voice wake-up method in the embodiment of the present application is further described. Figure 8 A flow chart of a voice wake-up method provided in an embodiment of the present application is shown. Figure 8 As shown:

[0229] The electronic device may include a wake-up detection module, a wake-up judgment module, a speech recognition module, and a semantic understanding module. Among them, the wake-up detection module can determine whether the user has a speech interaction intention based on motion detection, face detection, and ultrasonic distance detection; the wake-up judgment module can be used to determine whether the speech signal is a close-up speech and the direction of the sound source of the speech signal; the speech recognition module can be used to convert the speech signal into text information; and the semantic understanding module can be used to determine whether the text information includes a command word.

[0230] Exemplarily, in S801, the sensor collects IMU data, the camera collects images, and the ultrasonic module collects ultrasonic signals.

[0231] S802. The wake-up detection module outputs a first confidence level according to the IMU data, outputs a second confidence level according to the image captured by the camera, and outputs a third confidence level according to the ultrasonic signal.

[0232] S803: When the first confidence level, the second confidence level and / or the third confidence level meet the second preset condition, the electronic device may set the value of the first flag bit to the first value.

[0233] Steps S801-S803 are used for determination, and reference may be made to the relevant descriptions of steps S401-S404, which will not be repeated here.

[0234] Alternatively, when the first confidence level, the second confidence level and / or the third confidence level do not satisfy the second preset condition, the electronic device may set the value of the first flag bit to the second value, and the electronic device executes step S801.

[0235] S804: When the first flag bit is the first value, the electronic device turns on the voice enhancement algorithm module and the microphone.

[0236] S805: The microphone collects original voice signals.

[0237] The original voice signal may include an original voice signal of a bottom microphone, an original voice signal of a top microphone, and an original voice signal of a back microphone.

[0238] S806: The speech enhancement algorithm module processes the original speech signal to obtain a speech signal.

[0239] The voice signal may include a voice signal of a bottom microphone, a voice signal of a top microphone, and a voice signal of a back microphone.

[0240] S807: The wake-up determination module determines whether the audio feature of the speech signal meets a first preset condition based on the audio feature of the speech signal.

[0241] S808: When the audio feature of the voice signal meets the first preset condition, the wake-up judgment module obtains the sound source angle of the sound source based on the voice signal.

[0242] In step S808, the voice signal may be a voice signal of the bottom microphone and a voice signal of the top microphone.

[0243] Alternatively, when the audio feature of the speech signal does not meet the first preset condition, the electronic device executes step S801.

[0244] S809: When the sound source angle of the sound source is less than or equal to the preset angle, the electronic device sets the value of the second flag bit to a third value.

[0245] Alternatively, when the sound source angle of the sound source is greater than the preset angle, the electronic device executes step S801.

[0246] For steps S804-S809, reference may be made to the relevant descriptions of steps S501-S505, which will not be repeated here.

[0247] S810: When the second flag bit is a third value, the speech recognition module converts the speech signal into text information.

[0248] In step S810, the voice signal may be a voice signal of the bottom microphone.

[0249] S811. The semantic understanding module determines whether the text information is a command word.

[0250] S812: If the text message is a command word, the electronic device wakes up the voice assistant.

[0251] Alternatively, if the text information is a non-command word, the electronic device executes step S801.

[0252] For steps S810-S812, please refer to the relevant description of the semantic understanding module and will not be repeated here.

[0253] Based on the above embodiments, the present application embodiment provides a voice wake-up method. For example, Fig. 9 A flowchart of a voice wake-up method provided in an embodiment of the present application.

[0254] like Fig. 9 As shown: The voice wake-up method is applied to an electronic device, and the voice wake-up method may include the following steps:

[0255] S901. At a first moment, the electronic device receives a first voice, and the electronic device wakes up a voice assistant.

[0256] The keyword may be a word group, a phrase or a sentence. The keyword in the first voice may be a command word. For example, the keyword in the first voice is "play music". When "play music" is received, the electronic device may wake up the voice assistant and start the music application.

[0257] S902: At a second moment, the electronic device receives a second voice, and the electronic device wakes up the voice assistant; the keywords in the second voice are different from the keywords in the first voice.

[0258] The keyword in the second voice may be a command word, and the keyword in the second voice is different from the keyword in the first voice. For example, the keyword in the second voice is "call friend A". Upon receiving "call friend A", the electronic device wakes up the voice assistant and initiates a call to the electronic device held by user A.

[0259] The keyword in the second voice may also be a wake-up word, and the wake-up word is different from the command word in the first voice. For example, the second voice is "Hello, Xiao X (nickname of the electronic device)", and the electronic device wakes up the voice assistant.

[0260] It is understandable that in the embodiments of the present application, the electronic device can be used in scenarios without wake-up words or in scenarios including wake-up words. For example, when the user says the wake-up word, the electronic device can wake up the voice assistant; for another example, when the user says the command word, the electronic device uses the voice wake-up method provided in the embodiments of the present application to wake up the voice assistant. The embodiments of the present application do not limit this.

[0261] S903. At a third moment, the electronic device receives a third voice, and the electronic device does not wake up the voice assistant; the keywords in the third voice are different from the keywords in the first voice, and the keywords in the third voice are different from the keywords in the second voice.

[0262] The keywords in the third voice may be non-command words. The keywords in the third voice are different from the keywords in the first voice, and the keywords in the third voice are different from the keywords in the second voice. For example, the third voice may be a chat between the user and a friend, "I had lunch at school today, and it tasted good." At this time, the electronic device will not wake up the voice assistant.

[0263] It should be noted that, in the embodiment of the present application, the first moment may be earlier than the second moment, and the second moment may be earlier than the third moment. The first to third moments may also have other time sequences, which are not limited in the embodiment of the present application.

[0264] In the embodiment of the present application, the electronic device can determine whether to wake up the voice assistant based on the keyword in the voice signal; when the keyword is a command word, the electronic device wakes up the voice assistant; when the keyword is not a command word, the electronic device does not wake up the voice assistant. In this way, the wake-up scenarios of the voice interaction function of the electronic device can be enriched, and at the same time, the probability of erroneously waking up the voice assistant can be reduced.

[0265] Optional, in Fig. 9 Based on the illustrated embodiment, the keywords in the first voice satisfy any one or more of the following part-of-speech combinations: a verb and a noun combination, a noun and an adjective combination, or a noun and an interrogative word combination; the keywords in the second voice satisfy any one or more of the following part-of-speech combinations: a verb and a noun combination, a noun and an adjective combination, or a noun and an interrogative word combination; the keywords in the third voice do not satisfy any one or more of the following part-of-speech combinations: a verb and a noun combination, a noun and an adjective combination, or a noun and an interrogative word combination.

[0266] The part-of-speech combination can refer to the relevant description in the above embodiment for determining whether a voice signal is a command word; wherein, the keyword in the first voice is a command word, the keyword in the second voice is a command word, and the keyword in the third voice is a non-command word.

[0267] Optional, in Fig. 9 Based on the illustrated embodiment, the electronic device receives a first voice, and the electronic device wakes up the voice assistant, including:

[0268] The electronic device obtains a first voice; the electronic device converts the first voice into a first text; the electronic device extracts text features of the first text; a semantic understanding model of the electronic device determines, based on the text features, that keywords in the first voice can wake up a voice assistant; wherein the text features include the number of words in the first text, the content of the first text and / or the word vector of the first text; the electronic device wakes up the voice assistant.

[0269] Among them, the first voice can correspond to Figure 4-Figure 8 The speech signal in the illustrated embodiment; the first text may correspond to Figure 8 Text information in the illustrated embodiment.

[0270] Optional, in Fig. 9 Based on the illustrated embodiment, at a first moment, the distance between the sound source of the first voice and the electronic device is a first distance; at a second moment, the distance between the sound source of the second voice and the electronic device is the first distance; at a third moment, the distance between the sound source of the third voice and the electronic device is the first distance; the method also includes: at a fourth moment, the electronic device receives the second voice, and the electronic device does not wake up the voice assistant; at the fourth moment, the distance between the sound source of the second voice and the electronic device is a second distance, and the second distance is greater than the first distance.

[0271] Among them, the first distance can be understood as: the distance that can satisfy the execution of the voice wake-up method in the embodiment of the present application; the second distance can be understood as: the distance that cannot satisfy the execution of the voice wake-up method in the embodiment of the present application. The keyword in the second voice can be a command word. For example, at the fourth moment, although the keyword of the second voice is the command word "call friend A", the distance between the electronic device and the user is far, and the electronic device will not wake up the voice assistant.

[0272] It should be noted that the first distance and the second distance may not be actual values, and the electronic device may reflect the first distance and the second distance through methods such as user motion detection, face detection, ultrasonic distance detection, close-talking voice detection, and sound source angle detection in the above embodiments. This embodiment of the application does not limit this.

[0273] Optional, in Fig. 9 On the basis of the illustrated embodiment, after the electronic device receives the first voice, it also includes: when the audio feature of the first voice meets the first preset condition, the electronic device calculates the sound source angle of the sound source of the first voice; wherein the audio feature includes phase and / or audio energy; the sound source angle is: the angle between the line segment from the sound source of the first voice to the center point of the electronic device and the line segment from the center point of the electronic device to the first microphone; the first microphone is the microphone on the side of the electronic device away from the front camera; when the sound source angle is less than or equal to the preset angle, the electronic device executes the process of determining whether the keywords in the first voice can wake up the voice assistant.

[0274] The first microphone may correspond to the bottom microphone.

[0275] Optional, in Fig. 9 Based on the embodiment shown, the electronic device further includes a second microphone, which is a microphone on a side of the electronic device close to the front camera; the sound source angle θ satisfies the following formula: θ=acosd(τ au ×c / d); where τ auis the difference between the time when the first microphone obtains the first speech and the time when the second microphone obtains the first speech, c is the speed of sound, d is the distance from the first microphone to the second microphone, and acosd is the arc cosine sign.

[0276] The second microphone may correspond to the top microphone.

[0277] Optional, in Fig. 9 Based on the illustrated embodiment, the first preset condition includes one or more of the following: the phase difference between the first phase and the second phase is greater than a phase threshold; wherein the first phase is the phase of the first speech obtained by the first microphone, and the second phase is the phase of the first speech obtained by the second microphone; and / or, the energy difference between the first audio energy and the second audio energy is greater than an energy threshold; wherein the first audio energy is the audio energy of the first speech obtained by the first microphone, and the second audio energy is the audio energy of the first speech obtained by the second microphone.

[0278] Optional, in Fig. 9 On the basis of the illustrated embodiment, before the electronic device receives the first voice, it includes: when the confidence meets the second preset condition, the electronic device turns on the microphone and the voice enhancement algorithm; wherein the confidence is used to reflect the probability of the user having voice interaction with the electronic device, and the confidence is related to the inertial measurement unit IMU data of the electronic device, the image captured by the camera and / or the distance measured by the ultrasonic module; the microphone of the electronic device captures the first original voice; the electronic device receives the first voice, including: the electronic device processes the first original voice based on the voice enhancement algorithm to obtain the first voice.

[0279] Among them, the first original voice may correspond to Figure 4 and Figure 5 The original speech signal in the embodiment shown

[0280] Optional, in Fig. 9On the basis of the illustrated embodiment, the confidence level satisfies a second preset condition, including: the first confidence level is greater than a first threshold of the first confidence level; and / or, the second confidence level is greater than a first threshold of the second confidence level; and / or, the third confidence level is greater than a first threshold of the third confidence level; and / or, the first confidence level is greater than a second threshold of the first confidence level, and the second confidence level is greater than a second threshold of the second confidence level; and / or, the first confidence level is greater than a second threshold of the first confidence level, and the third confidence level is greater than a second threshold of the third confidence level; and / or, the second confidence level is greater than a second threshold of the second confidence level, and the third confidence level is greater than a second threshold of the third confidence level; wherein, the first confidence level is obtained by the electronic device based on IMU data; the second confidence level is obtained by the electronic device based on the proportion of the face in the image; the third confidence level is obtained by the electronic device based on the distance between the user and the ultrasonic module; the first threshold of the first confidence level is greater than the second threshold of the first confidence level; the first threshold of the second confidence level is greater than the second threshold of the second confidence level; the first threshold of the third confidence level is greater than the second threshold of the third confidence level.

[0281] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0282] Combined with the above Figure 1-Figure 9 , the voice wake-up method provided in the embodiment of the present application is described, and the device for executing the voice wake-up method provided in the embodiment of the present application is described below. Fig.10 As shown, Fig.10 This is a structural schematic diagram of a voice wake-up device provided in an embodiment of the present application. The voice wake-up device 1000 can be a terminal device in an embodiment of the present application, or a chip or chip system in a terminal device.

[0283] like Fig.10As shown, the voice wake-up device 1000 can be used in a communication device, circuit, hardware component or chip, and the voice wake-up device 1000 includes: a memory 1001, a processor 1002, an interface circuit 1003 and a display screen 1004. Among them, the display screen 1004 is used to support the display step executed by the voice wake-up method; the processor 1002 is used to support the voice wake-up device to execute the information processing step, and the interface circuit 1003 is used to support the voice wake-up device to execute the receiving or sending step; the memory 1001 is used to support the voice wake-up device to execute the storage step. The display screen 1004 can also be called a display unit; the processor 1002 can also be called a processing unit, the interface circuit 1003 can also be called a communication unit; the memory 1001 can also be called a storage unit.

[0284] In one possible implementation, the voice wake-up device 1000 is used: at a first moment, the processor 1002 receives a first voice, and the processor 1002 wakes up the voice assistant; at a second moment, the processor 1002 receives a second voice, and the processor 1002 wakes up the voice assistant; the keywords in the second voice are different from the keywords in the first voice; the first moment is earlier than the second moment; at a third moment, the processor 1002 receives a third voice, and the processor 1002 does not wake up the voice assistant; the keywords in the third voice are different from the keywords in the first voice, and the keywords in the third voice are different from the keywords in the second voice; the second moment is earlier than the third moment.

[0285] In one possible implementation, the keywords in the first voice satisfy any one or more of the following part-of-speech combinations: a verb and a noun combination, a noun and an adjective combination, or a noun and an interrogative word combination; the keywords in the second voice satisfy any one or more of the following part-of-speech combinations: a verb and a noun combination, a noun and an adjective combination, or a noun and an interrogative word combination; the keywords in the third voice do not satisfy any one or more of the following part-of-speech combinations: a verb and a noun combination, a noun and an adjective combination, or a noun and an interrogative word combination.

[0286] In one possible implementation, the voice wake-up device 1000 is specifically used for: the processor 1002 obtains a first voice; the processor 1002 converts the first voice into a first text; the processor 1002 extracts text features of the first text; the semantic understanding model of the processor 1002 determines that keywords in the first voice can wake up the voice assistant based on the text features; wherein the text features include the number of words in the first text, the content of the first text and / or the word vector of the first text; the processor 1002 wakes up the voice assistant.

[0287] In one possible implementation, at a first moment, the distance between the sound source of the first voice and the voice wake-up device 1000 is the first distance; at a second moment, the distance between the sound source of the second voice and the voice wake-up device 1000 is the first distance; at a third moment, the distance between the sound source of the third voice and the voice wake-up device 1000 is the first distance; the voice wake-up device can also be used for: at a fourth moment, the processor 1002 receives the second voice, and the processor 1002 does not wake up the voice assistant; the third moment is later than the fourth moment; at the fourth moment, the distance between the sound source of the second voice and the voice wake-up device 1000 is the second distance, and the second distance is greater than the first distance.

[0288] In one possible implementation, after the processor 1002 receives the first voice, the voice wake-up device 1000 is also used to: when the audio feature of the first voice meets the first preset condition, the processor 1002 calculates the sound source angle of the sound source of the first voice; wherein the audio feature includes phase and / or audio energy; the sound source angle is: the angle between the line segment from the sound source of the first voice to the center point of the voice wake-up device 1000 and the line segment from the center point of the voice wake-up device 1000 to the first microphone; the first microphone is the microphone on the side of the voice wake-up device 1000 away from the front camera; when the sound source angle is less than or equal to the preset angle, the processor 1002 executes a process for determining whether the keywords in the first voice can wake up the voice assistant.

[0289] In a possible implementation, the voice wake-up device 1000 further includes a second microphone, which is a microphone on a side of the voice wake-up device 1000 close to the front camera; the sound source angle θ satisfies the following formula: θ=acosd(τ au ×c / d); where τ au is the difference between the time when the first microphone obtains the first speech and the time when the second microphone obtains the first speech, c is the speed of sound, d is the distance from the first microphone to the second microphone, and acosd is the arc cosine sign.

[0290] In one possible implementation, the first preset condition includes one or more of the following: a phase difference between a first phase and a second phase is greater than a phase threshold; wherein the first phase is the phase of the first speech obtained by the first microphone, and the second phase is the phase of the first speech obtained by the second microphone; and / or an energy difference between the first audio energy and the second audio energy is greater than an energy threshold; wherein the first audio energy is the audio energy of the first speech obtained by the first microphone, and the second audio energy is the audio energy of the first speech obtained by the second microphone.

[0291] In one possible implementation, before the processor 1002 receives the first voice, the voice wake-up device 1000 is used to: when the confidence meets the second preset condition, the processor 1002 turns on the microphone and the voice enhancement algorithm; wherein the confidence is used to reflect the probability of the user performing voice interaction with the voice wake-up device 1000, and the confidence is related to the inertial measurement unit IMU data, the image captured by the camera and / or the distance measured by the ultrasonic module; the microphone of the voice wake-up device 1000 collects the first original voice; the processor 1002 receives the first voice, including: the processor 1002 processes the first original voice based on the voice enhancement algorithm to obtain the first voice.

[0292] In a possible implementation, the confidence level satisfies a second preset condition, including: the first confidence level is greater than a first threshold of the first confidence level; and / or, the second confidence level is greater than a first threshold of the second confidence level; and / or, the third confidence level is greater than a first threshold of the third confidence level; and / or, the first confidence level is greater than a second threshold of the first confidence level, and the second confidence level is greater than a second threshold of the second confidence level; and / or, the first confidence level is greater than a second threshold of the first confidence level, and the third confidence level is greater than a second threshold of the third confidence level; and / or, the second confidence level is greater than a second threshold of the second confidence level, and the third confidence level is greater than a second threshold of the third confidence level; wherein, the first confidence level is obtained by the processor 1002 based on IMU data; the second confidence level is obtained by the processor 1002 based on the proportion of the face in the image; the third confidence level is obtained by the processor 1002 based on the distance between the user and the ultrasonic module; the first threshold of the first confidence level is greater than the second threshold of the first confidence level; the first threshold of the second confidence level is greater than the second threshold of the second confidence level; the first threshold of the third confidence level is greater than the second threshold of the third confidence level.

[0293] The voice wake-up method provided in the embodiment of the present application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can refer to the above related descriptions, which will not be repeated here.

[0294] An embodiment of the present application provides a terminal device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.

[0295] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0296] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0297] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0298] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0299] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0300] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A voice wake-up method, characterized in that: Used in electronic equipment, including: At a first moment, the electronic device receives a first voice, and the electronic device wakes up the voice assistant; At a second moment, the electronic device receives a second voice, and the electronic device wakes up the voice assistant; the keywords in the second voice are different from the keywords in the first voice; the first moment is earlier than the second moment; At a third moment, the electronic device receives a third voice, and the electronic device does not wake up the voice assistant; the keywords in the third voice are different from the keywords in the first voice, and the keywords in the third voice are different from the keywords in the second voice; the second moment is earlier than the third moment; Before the electronic device receives the first voice, the method includes: When the confidence level meets the second preset condition, the electronic device turns on the microphone and the voice enhancement algorithm; wherein the confidence level is used to reflect the probability of the user performing voice interaction with the electronic device, and the confidence level is related to the inertial measurement unit IMU data of the electronic device, the image collected by the camera, and the distance measured by the ultrasonic module; The microphone of the electronic device collects a first original voice; The electronic device receives a first voice, including: The electronic device processes the first original speech based on the speech enhancement algorithm to obtain the first speech; After the electronic device receives the first voice, the method further includes: When the audio feature of the first voice satisfies a first preset condition, the electronic device calculates the sound source angle of the sound source of the first voice; wherein the audio feature includes a phase and / or audio energy; the sound source angle is: the angle between a line segment from the sound source of the first voice to the center point of the electronic device and a line segment from the center point of the electronic device to a first microphone; the first microphone is a microphone on a side of the electronic device away from a front camera; When the sound source angle is less than or equal to a preset angle, the electronic device executes a process of determining whether a keyword in the first voice can wake up the voice assistant.

2. The method according to claim 1, characterized in that The keywords in the first speech satisfy any one or more of the following part-of-speech combinations: a combination of a verb and a noun, a combination of a noun and an adjective, or a combination of a noun and a question word; The keywords in the second speech satisfy any one or more of the following part-of-speech combinations: a combination of a verb and a noun, a combination of a noun and an adjective, or a combination of a noun and a question word; The keywords in the third speech do not satisfy any one or more of the following part-of-speech combinations: a combination of a verb and a noun, a combination of a noun and an adjective, or a combination of a noun and a question word.

3. The method according to claim 1 or 2, characterized in that: The electronic device receives a first voice, and the electronic device wakes up a voice assistant, including: The electronic device obtains the first voice; The electronic device converts the first speech into a first text; The electronic device extracts text features of the first text; The semantic understanding model of the electronic device determines, based on the text features, that a keyword in the first voice can wake up the voice assistant; wherein the text features include the number of words in the first text, the content of the first text and / or the word vector of the first text; The electronic device wakes up the voice assistant.

4. The method according to any one of claims 1 to 3, characterized in that: At the first moment, the distance between the sound source of the first voice and the electronic device is a first distance; at the second moment, the distance between the sound source of the second voice and the electronic device is the first distance; At the third moment, the distance between the sound source of the third voice and the electronic device is the first distance; and the method further includes: At a fourth moment, the electronic device receives the second voice, and the electronic device does not wake up the voice assistant; the third moment is later than the fourth moment; At the fourth moment, the distance between the sound source of the second voice and the electronic device is a second distance, and the second distance is greater than the first distance.

5. The method according to claim 1, characterized in that The electronic device further includes a second microphone, which is a microphone on a side of the electronic device close to the front camera; the sound source angle θ satisfies the following formula: θ=acosd(τ au ×c / d) Among them, τ au is the difference between the time when the first microphone obtains the first speech and the time when the second microphone obtains the first speech, c is the speed of sound, d is the distance from the first microphone to the second microphone, and acosd is the arc cosine sign.

6. The method according to claim 5, characterized in that The first preset condition includes one or more of the following: The phase difference between the first phase and the second phase is greater than a phase threshold; wherein the first phase is the phase of the first speech obtained by the first microphone, and the second phase is the phase of the first speech obtained by the second microphone; And / or, an energy difference between the first audio energy and the second audio energy is greater than an energy threshold; wherein the first audio energy is the audio energy of the first speech obtained by the first microphone, and the second audio energy is the audio energy of the first speech obtained by the second microphone.

7. The method according to claim 1, characterized in that The confidence level satisfies a second preset condition, including: The first confidence level is greater than a first threshold of the first confidence level; and / or, the second confidence level is greater than a first threshold value of the second confidence level; and / or, the third confidence level is greater than a first threshold of the third confidence level; and / or, the first confidence level is greater than a second threshold of the first confidence level, and the second confidence level is greater than a second threshold of the second confidence level; And / or, the first confidence level is greater than a second threshold of the first confidence level, and the third confidence level is greater than the second threshold of the third confidence level; And / or, the second confidence level is greater than a second threshold of the second confidence level, and the third confidence level is greater than the second threshold of the third confidence level; Among them, the first confidence level is obtained by the electronic device based on the IMU data; the second confidence level is obtained by the electronic device based on the proportion of the face in the image; the third confidence level is obtained by the electronic device based on the distance between the user and the ultrasonic module; the first threshold of the first confidence level is greater than the second threshold of the first confidence level; the first threshold of the second confidence level is greater than the second threshold of the second confidence level; the first threshold of the third confidence level is greater than the second threshold of the third confidence level.

8. A terminal device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The method comprises a computer program, which, when being executed, enables a computer to execute the method according to any one of claims 1 to 7.

Images