Voice wake-up method and electronic equipment

By identifying and splicing wake words in the voice information flow, ensuring that they are in the correct position of the buffer, the problem that electronic devices cannot successfully wake up the voice assistant when they wake up voice is solved, realizing the one-word direct mode, improving user experience and device Interaction efficiency.

CN119049459AActive Publication Date: 2024-11-29HONOR DEVICE CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202410917677.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-29
Estimated Expiration
2044-07-09

AI Technical Summary

Technical Problem

When the electronic device wakes up voice, it cannot successfully wake up the voice assistant application or fails to enter the one-word direct mode, affecting the user experience.

Method used

By identifying and splicing the voice information flow containing wake-up words, ensure that the wake-up words are in the first 2 seconds of the buffer, the data structure and voiceprint recognition module of the ring buffer are used to achieve complete recognition of wake-up words, and after successful recognition, the data structure of the ring buffer is used. Enter the one-word direct mode.

Benefits of technology

The voice assistant application is successfully awakened and the voice command is directly responded to the voice command after recognizing the wake-up word, which improves the user experience and the interaction efficiency of the device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049459A_ABST
    Figure CN119049459A_ABST
Patent Text Reader

Abstract

The invention provides a voice wake-up method and electronic equipment, and relates to the technical field of electronic equipment, and the method comprises the steps that the electronic equipment receives a voice information stream input by a user, the voice information stream is divided into multiple pieces of voice information of first duration, and first voice information including a wake-up word exists in the multiple pieces of voice information of the first duration; and the electronic equipment performs wake-up word recognition on the voice information of the second duration including the wake-up word in the first voice information. Then, in response to the situation that the electronic equipment recognizes that the wake-up word exists in the voice information of the second duration, the electronic equipment carries out voice instruction recognition on the second voice information in the voice information of the multiple first durations; the second voice information is voice information input by the user after the first voice information in the voice information flow; the electronic equipment responds to the voice instruction recognized from the second voice information, and operation corresponding to the voice instruction is executed. According to the scheme, the voice assistant application can be awakened, and one-language-through can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of electronic equipment, and in particular to a voice wake-up method and electronic equipment. Background Art

[0002] At present, with the development of artificial intelligence, voice assistant applications of electronic devices provide more convenience for users. For example, users can control electronic devices to open applications, play music, etc. through voice assistant applications, and can also have conversations and chats with voice assistant applications. Usually, when users control electronic devices through voice assistant applications, they can wake up the voice assistant applications through fixed words.

[0003] However, there may be situations where the electronic device cannot wake up the voice assistant application. Summary of the invention

[0004] The embodiments of the present application provide a voice wake-up method and an electronic device, which can successfully wake up a voice assistant application and achieve one-sentence direct access.

[0005] In the first aspect, a voice wake-up method is provided, which can be applied to an electronic device, which is installed with a voice assistant application. The electronic device can receive a voice information stream input by a user, and the voice information stream is divided into a plurality of voice information of a first duration, and there is a first voice information including a wake-up word in the plurality of voice information of the first duration, and the wake-up word is used to wake up the voice assistant application. The voice information stream input by the user is a segment of audio data input by the user. The electronic device recognizes the wake-up word for the voice information of the second duration in the first voice information; wherein the first duration is greater than the second duration, and the voice information of the second duration includes the wake-up word; in response to the electronic device recognizing the presence of the wake-up word in the voice information of the second duration, the electronic device recognizes the voice command for the second voice information in the plurality of voice information of the first duration; the second voice information is the voice information input by the user after the first voice information in the voice information stream. In response to recognizing the voice command from the second voice information, the electronic device executes the operation corresponding to the voice command. Since the electronic device only recognizes the voice information of the second duration when detecting the wake-up word, the electronic device can recognize the selected voice information of the second duration. Furthermore, since the voice information of the second duration includes the wake-up word, the electronic device can ensure that the wake-up word is detected when detecting the wake-up word in the voice information of the second duration.

[0006] When the electronic device recognizes the wake-up word in the voice information of the second duration, if the voiceprint recognition of the wake-up word is successful, the electronic device can successfully wake up the voice assistant application. Then, at this time, the electronic device can continue to recognize the second voice information in the voice information stream. The second voice information is the voice information input by the user after the first voice information. Therefore, after the electronic device successfully wakes up the voice assistant application, if there is sound in the second voice information that continues to be detected, then the electronic device successfully turns on the one-word direct access function at this time, and can directly respond to the voice command after the wake-up word. In response to recognizing the voice command from the second voice information, the electronic device executes the operation corresponding to the voice command.

[0007] By adopting this technical solution, when the electronic device is voice-wake up, it only recognizes the selected voice information containing the wake-up word. After the electronic device detects the wake-up word and the voiceprint recognition is successful, it can successfully wake up the voice assistant. And, since the voice information detected subsequently is audible, the electronic device can directly respond to the detected voice command to perform the corresponding operation. As a result, the electronic device can successfully wake up the voice assistant when it receives the user's wake-up word and voice command, and directly perform the corresponding operation according to the recognized voice command, thereby improving the user experience.

[0008] In a possible implementation of the first aspect, the electronic device includes a buffer, which is used to store the first voice information. The voice information of the second duration is the voice information of the first preset number of frames starting from the first frame in the first voice information stored in the buffer, and the first preset number of frames corresponds to the second duration. The second preset number of frames here corresponds to 300 milliseconds. That is to say, after waking up the voice assistant application, the electronic device continues to detect whether the audio data of the first 300 milliseconds of the remaining audio data is sound data, and it can successfully enter the one-speech direct access.

[0009] In a possible implementation of the first aspect, the electronic device does not recognize voice information other than the voice information of the second duration in the first voice information stored in the buffer. After the electronic device recognizes the wake-up word in the voice information of the second duration stored in the buffer, if the voiceprint recognition is successful, it will not recognize voice information other than the voice information of the second duration in the buffer, but will continue to recognize the second voice information. Since the electronic device detects that there is sound in the first 300 milliseconds of the second voice information after successfully recognizing the voiceprint of the wake-up word, the one-speech direct access function can be successfully enabled.

[0010] In a possible implementation of the first aspect, the electronic device includes a system control processor and a voiceprint recognition module; the electronic device performs wake-up word recognition on the voice information of the second duration in the first voice information, including: the system control processor of the electronic device intercepts the voice information of the second duration from the first voice information stored in the buffer; the system control processor of the electronic device sends the voice information of the second duration to the voiceprint recognition module; the voiceprint recognition module of the electronic device performs wake-up word recognition on the voice information of the second duration. Thus, the system control processor of the electronic device only sends the voice information of the second duration to the voiceprint recognition module, so as to ensure that the voiceprint recognition module recognizes the complete wake-up word when performing wake-up word recognition.

[0011] In a possible implementation of the first aspect, the buffer includes a head pointer and a tail pointer. Before the electronic device recognizes the wake-up word for the voice information of the second duration in the first voice information, it includes: the electronic device identifies the position of the wake-up word from the buffer and places the tail pointer at the tail of the wake-up word. The electronic device moves the position of the tail pointer by a third preset frame number; the third preset frame number is the number of frames corresponding to the difference between the first duration and the second duration; the electronic device moves the voice information corresponding to the first area of ​​the buffer to the front of the voice information corresponding to the second area of ​​the buffer, and splices it with the voice information of the second area to obtain the spliced ​​first voice information; the first area is the area between the position after the tail pointer is moved and the end position of the buffer, and the second area is the area between the position of the head pointer and the position after the tail pointer is moved; the head pointer points to the starting position of the buffer, and the end position of the buffer points to the position of the head pointer. Thus, after splicing, the voice information where the wake-up word is located is at the position of the first second duration of the buffer. The electronic device can directly select the audio data of the first second duration of the buffer for voiceprint recognition. Since the audio data of the first second duration includes the complete wake-up word, after the voiceprint recognition is successful, the voice assistant application can be successfully woken up.

[0012] In a possible implementation of the first aspect, the electronic device shifts the position of the tail pointer by a third preset number of frames, including: the electronic device shifts the tail pointer in a first direction, the first direction being the direction pointing to the end position of the buffer; if the shifted frame number is equal to the third preset frame number before the tail pointer is shifted to the end position of the buffer, the shifting is terminated. If the shifted frame number is less than the third preset frame number when the tail pointer is shifted to the end position of the buffer, the tail pointer continues to be shifted in the first direction starting from the position of the head pointer until the shifted frame number is equal to the third preset frame number. The above shifting of the position of the tail pointer is to ensure that after the audio data in the buffer is shifted and spliced, the first second time length of the buffer includes a complete wake-up word, so that the electronic device can wake up the voice assistant application after the voiceprint recognition is successful.

[0013] In a second aspect, the present application provides an electronic device, comprising: a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; wherein the memory stores computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the processor, the electronic device executes any one of the methods described in the first aspect above.

[0014] In a third aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer can execute any of the methods described in the first aspect.

[0015] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect.

[0016] It can be understood that the electronic device described in the second aspect provided above, the computer-readable storage medium described in the third aspect and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of an interface for enabling a voice wake-up function provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of an interface for enabling a one-word direct access function provided in an embodiment of the present application;

[0019] Figure 3 A specific flow chart of voice wake-up provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of an audio dump log provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of the system architecture of a voice wake-up method provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of shifting and splicing audio data in a first buffer provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of another audio data shifting and splicing in a first buffer provided in an embodiment of the present application;

[0024] Figure 8A schematic diagram of shifting and splicing audio data in a first buffer provided in another embodiment of the present application;

[0025] Fig. 9 A schematic diagram of audio data in a ring buffer provided in an embodiment of the present application;

[0026] Fig.10 A schematic diagram of audio data in a ring buffer provided in an embodiment of the present application;

[0027] Fig.11 A schematic diagram of a specific flow chart for a direct access to an embodiment of the present application;

[0028] Fig.12 A schematic diagram of another audio dump log provided in an embodiment of the present application;

[0029] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0030] Fig.14 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0032] It should be noted that the following terms "first", "second", etc. are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0033] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0034] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0035] In order to better illustrate the technical solution of the present application, the following is a brief overview of the terms involved in the present application.

[0036] (1) Voice assistant application: An intelligent voice interaction system based on artificial intelligence technology that can communicate with users through voice to help users complete various operations or provide various services. Voice assistant applications are usually built into smart electronic devices such as smartphones, smart speakers, and smart watches. Users can use voice commands to perform various functions, such as checking the weather, playing music, sending text messages, setting alarms, etc.

[0037] The working principle of voice assistant applications is to recognize the user's voice commands as text, then use natural language processing technology to analyze the text, understand the user's intentions, and finally provide corresponding services according to user needs. The core technologies of voice assistant applications include speech recognition, natural language processing, speech synthesis, etc.

[0038] Among them, the user can operate the physical buttons of the electronic device, such as long pressing the power button of the electronic device to wake up the voice assistant application, or can also wake up the voice assistant application by voice wake-up.

[0039] (2) Voice wake-up: A human-computer interaction technology used to wake up the voice assistant application in an electronic device. Through this technology, the user can activate the electronic device through specific voice information (usually a word or phrase) to start its voice assistant, allowing the user to interact with the electronic device through voice commands.

[0040] The principle of voice wake-up is to extract features from the input voice information, and then compare the extracted features with the pre-stored feature templates to determine whether the input voice information is a specific word. If it is a specific word, the voice assistant application in the electronic device is woken up. If it is not a specific word, the voice assistant application in the electronic device will not be woken up.

[0041] The user wakes up the voice assistant application by voice wake-up, which may be by saying a wake-up word.

[0042] See also Figure 1 , Figure 1 Schematic diagram of an interface for enabling the voice wake-up function. Taking a mobile phone as an example, the mobile phone is provided with a smart assistant application. The smart assistant application can be understood as a voice assistant application. Figure 1 As shown in (a), the mobile phone can respond to the user's selection operation of the control 102 corresponding to the smart assistant application in the setting interface 101, and display the following Figure 1 The smart assistant application interface 103 shown in (b) of FIG. The smart assistant application interface 103 may display a wake-up word 104 corresponding to voice wake-up, such as "Hello xx". The user may also customize the wake-up word.

[0043] The smart assistant application interface 103 also includes a voice assistant control 105. The mobile phone can respond to the user's selection operation of the voice assistant control 105 and display the following information: Figure 1 The voice assistant interface 106 shown in (c) of FIG. 106 includes a control 107 corresponding to voice wake-up. At this time, the voice wake-up function is not turned on, and the control 107 corresponding to voice wake-up displays the word "off". The mobile phone can respond to the user's selection operation of the control 107 corresponding to voice wake-up by displaying the following Figure 1 The voice wake-up interface 108 shown in (d).

[0044] The voice wake-up interface 108 includes a switch control 109 corresponding to the voice wake-up. At this time, the switch control 109 is in a closed state. The mobile phone can respond to the user's opening operation of the switch control 109 and display the following Figure 1 The voice wake-up interface 108 shown in (e) at this time, the switch control 109 is in the open state. Figure 1 As shown in (e), the voice wake-up interface 108 includes a control 110 corresponding to the wake-up word.

[0045] The mobile phone can respond to the user's selection operation of the control 110 corresponding to the wake-up word and enter the following Figure 1(f) shows the setting interface 111 of the wake-up word. In the setting interface 111 of the wake-up word, a prompt message 112 for inputting the wake-up word may be displayed, and the user may complete the input operation of the wake-up word according to the prompt message 112.

[0046] Therefore, after the mobile phone completes the user's input operation of the wake-up word, it can wake up the smart assistant application in response to the wake-up word entered by the user.

[0047] After the voice wake-up function is turned on, it can include two forms, one is ordinary voice wake-up, and the other is direct access. Among them, ordinary voice wake-up means that after the mobile phone recognizes the wake-up word entered by the user, it first responds to the wake-up word, and then the mobile phone interacts with the electronic device according to the voice command subsequently entered by the user. For example, the mobile phone recognizes the wake-up word "Hello xx" entered by the user, replies "I am here", and then the mobile phone continues to recognize the voice command entered by the user.

[0048] (3) Direct access: When waking up with voice, users can enter the wake-up word and voice command at the same time. There is no need to wait for the electronic device to respond to the wake-up word input by the user. The electronic device can directly respond to the voice command input by the user to interact with the user. For example, the user directly says "Hello xx, turn up the volume" to the mobile phone. After the voice assistant application wakes up, the mobile phone does not need to respond to the wake-up word "Hello xx", but directly performs the operation of turning up the volume.

[0049] See also Figure 2 , Figure 2 This is a schematic diagram of an interface that enables the one-word direct access function. Figure 1 ,like Figure 2 As shown in (a), the voice wake-up interface 108 includes a switch control 201 corresponding to the one-speech direct access function. At this time, the switch control 201 is in a closed state. The mobile phone can respond to the user's opening operation of the switch control 201 to turn on the one-speech direct access function in the voice wake-up. Figure 2 As shown in (b), the switch control 201 is in the on state, which means that the one-sentence direct access function has been turned on in the mobile phone.

[0050] (4) Ring buffer, also known as circular queue, circular buffer, circular buffer, is a data structure used to represent a fixed-size, head-to-tail connected buffer that can be used to cache data streams. The ring buffer is a first-in-first-out circular buffer. When initialized, the content of the ring buffer is empty and has a predetermined length.

[0051] The ring buffer is an array data structure that includes a "head pointer (head_ptr)" and a "tail pointer (tail_ptr)". The "head pointer" points to the buffer space in the ring buffer where data can be read, and the "tail pointer" points to the buffer space in the ring buffer where data can be written. Data in the ring buffer can be read and written by moving the "head pointer" and "tail pointer".

[0052] In the present embodiment, after the mobile phone receives the audio data input by the user, the audio data, i.e., the above-mentioned voice information, can use a ring buffer to store the audio data. Here, for the ring buffer storing audio data, one ring buffer can store 2.34 seconds (s) of audio data. For example, if the mobile phone receives 6 seconds of audio data input by the user, then 3 ring buffers can be used to store 6 seconds of audio data. Specifically, the first buffer stores the audio data from the 0th second to the 2.34th second, the second buffer stores the audio data from the 2.34th second to the 4.68th second, and the third buffer stores the audio data from the 4.68th second to the 6th second. Wherein, the first buffer and the second buffer are in a full state, and the third buffer is in an incomplete state.

[0053] It should be noted that the working principle of the circular buffer is to store all 0s after 2.34s are fully stored, and then store again. In fact, there is only one circular buffer in the mobile phone, and the above-mentioned first buffer, second buffer and third buffer are customized in this embodiment for the convenience of subsequent description. The mobile phone can divide a received audio data into multiple 2.34-second audio data according to the time sequence of receiving the audio data. The "X"th buffer here can be understood as the buffer storing the "X"th 2.34 seconds of audio data in a segment of audio data. For example, the first buffer stores the first 2.34 seconds of audio data in a segment of audio data, that is, the audio data from the 0th second to the 2.34th second.

[0054] In the scenario where the user wakes up the voice assistant application by voice wake-up, if the electronic device has turned on the one-word direct access function, after the electronic device receives the audio data including the wake-up word and the voice command, there may be a situation where the electronic device fails to successfully wake up the voice assistant application. Alternatively, there may be a situation where the electronic device successfully wakes up the voice assistant application but does not execute the voice command. Among them, the following explanation is based on the example of the electronic device being a mobile phone and the wake-up word being "Hello xx".

[0055] For example, the user says "Hello xx, turn up the volume", but the phone does not respond and does not perform the volume-up operation. At this point, even if the user enters a voice command again, such as "turn up the volume", the phone will not perform the volume-up operation. In the above situation, it can be considered that the phone fails to wake up the voice assistant application and needs to enter the wake-up word again, which affects the user experience.

[0056] Or, for example, if the user says "Hello xx, turn up the volume", the phone will not directly perform the volume-up operation, but will reply "I am here". The user needs to enter a voice command again, such as "turn up the volume". In response to the voice command received at this time, the phone performs the volume-up operation. In the above situation, it can be considered that the phone successfully wakes up the voice assistant application, but does not enter the one-word direct access, but enters the normal wake-up, and needs to receive the user's voice command again, which affects the user experience.

[0057] First, let's introduce the specific processes of normal wake-up and one-sentence direct access in voice wake-up.

[0058] See also Figure 3 , Figure 3 A specific flow chart of voice wake-up provided in the embodiment of the present application. Figure 3 As shown, the mobile phone can be divided into at least an application processor (AP) module and a system control processor (SCP) module. The SCP module includes a memory block where the wake-up word is located, and the memory block here can refer to a ring buffer. The SCP module can store audio data including the wake-up word through the ring buffer.

[0059] Under the Android architecture, the AP module can be divided into at least an application layer and a framework layer. The application layer includes the voice assistant application. The framework layer includes the voiceprint recognition module, the voice activity detection (VAD) module and the automatic speech recognition technology (ASR) module.

[0060] The specific process of voice wake-up is as follows: The SCP module can periodically detect the wake-up word, which can also be called a hot word. Once the SCP module detects the hot word, it can report a detection event to the voice assistant application. Among them, the detection event is a first-level wake-up event.

[0061] Then, the voice assistant application sends the first stream (first identification stream) notification after receiving the detect event. The first identification stream can be called source 1999. The first identification stream no longer records in real time through a microphone (mic), but directly obtains the audio data stored in the wake-up dedicated ring buffer of the SCP module. The wake-up dedicated ring buffer refers to the buffer containing the hotword.

[0062] The voice assistant application sends the audio data containing hot words from the locally cached recording, that is, from the audio data stored in the buffer of the SCP module, to the voiceprint recognition module. The voiceprint recognition module can perform voiceprint recognition on the received audio data containing hot words, and this voiceprint recognition can be a secondary wake-up event. If the voiceprint corresponding to the audio data is consistent with the voiceprint registered in the mobile phone, the voiceprint recognition is successful, otherwise the voiceprint recognition fails.

[0063] After the voiceprint recognition is successful, the voiceprint recognition module returns the hot word end node to the VAD module, and the VAD module then determines whether there is a speaking sound in the first 300 milliseconds (ms) of the remaining audio data after the hot word end node. The VAD module can obtain the audio data stored in the ring buffer of the SCP module.

[0064] If there is no speaking sound within 300ms, it can be judged as a normal wake-up. Then, the VAD module sends a notification message to the voice assistant application, which contains a message that this voice wake-up is a normal wake-up. Among them, the stream of source 6 is different from the stream of source 1999. The stream of source 6 is recorded in real time through the main microphone, and then the audio data is sent to the ASR module for command recognition. After the ASR module recognizes the command, it operates the mobile phone according to the command. After receiving the notification message, the voice assistant application sends a second stream (second recognition stream) notification. The second recognition stream can be called source 6. The second recognition stream is different from the first recognition stream and is recorded in real time through the main microphone.

[0065] If there is a speaking sound within 300ms, it can be judged as a one-shot wake-up. Then, the VAD module sends a notification message to the voice assistant application, which contains a message that this voice wake-up is a one-shot wake-up. After receiving the notification message, the voice assistant application sends the locally cached recording to the ASR module. The ASR module recognizes the instructions in the locally cached recording and operates the phone according to the instructions.

[0066] Therefore, even if the mobile phone has turned on the one-speech direct access function, after receiving the audio data entered by the user, if the voiceprint recognition is successful, and it continues to determine that there is no sound in the next 300ms of audio data, the mobile phone will not enter the one-speech direct access function, but will enter the normal wake-up mode.

[0067] In summary, the absence of speaking sound in the first 300ms of the remaining audio data is the reason why the above-mentioned mobile phone did not enter the one-speech direct access but entered the normal wake-up after successfully waking up the voice assistant application.

[0068] For the above-mentioned problem scenario where the mobile phone successfully wakes up the voice assistant application but does not enter the one-sentence direct access function, the mobile phone can capture the dump log of the problem audio. First, the mobile phone obtains the pulse code modulation (PCM) node file (PCM_FROM_HAL_RECORD node file) of the hardware abstraction layer (HAL) record (RECORD) from the dump log, and determines from the node file whether the audio data has been uploaded to the framework layer, and whether the uploaded audio data meets expectations. Among them, the node file can be opened with a 16K-16bit-1 channel. For the audio content corresponding to the node file, please refer to Figure 4 .like Figure 4 As shown in the figure, there is a silence frame of about 360ms between "Turn up" and "Volume". By understanding the judgment logic of the upper layer of one-speech direct access, it can be confirmed that this 360ms silence frame is the root cause of the failure of entering one-speech direct access.

[0069] The above can confirm that the problem is below the HAL layer. Then you can check the audio dump node file (StreamI n_Dump.pcm0.AudioALSACaptureDataProviderVOW.16000.16bit.1ch). Among them, you can use 16K-16bit-1 channel to open this file. The audio content corresponding to this node file can also be referred to Figure 4 .like Figure 4 As shown, there is a silent frame of about 360ms between "Turn Up" and "Volume", which is exactly the same as the audio content in the above PCM_FROM_HAL_RECORD node file.

[0070] From the above two node files, we can know that there is a problem with the audio of the HAL layer. However, currently, we cannot determine whether the problem is caused by the SCP module or the direct memory access (DMA) transfer from the SCP module to the AP module based on the dump log of the problem audio. For further accurate positioning, we can analyze the sound receiving logic, transfer logic and problem log of the SCP module wake-up ring buffer.

[0071] Then, if the above mobile phone successfully wakes up the voice assistant application, but does not enter the one-word direct access problem scenario, it means that the voice assistant application did not detect any sound when it detected the first 300 milliseconds of audio data in the subsequent audio data through the VAD module. In other words, the voice assistant application did not detect any sound when it continued to read the audio data in the ring buffer including the wake-up word after 2 seconds through the VAD module. Figure 4 Correspondingly, it can be considered that the last 340ms in the ring buffer including the wake-up word is a silent frame.

[0072] Among them, the two key functions corresponding to the ring buffer including the wake-up word in the SCP module are: recording process function (vow_record_process) and seamless recording function (vow_seamless_record). From the contents recorded by these two functions, it can be seen that the length of the ring buffer is not 2s required by the upper layer (such as the application layer), but 2.34s. There may be splicing of audio data in vow_seamless_record, and the audio stream content finally returned to the voice assistant application is the audio data in the 2.34s ring buffer. For example, if a buffer does not store 2.34s of audio data, it will be filled up before uploading to the voice assistant application.

[0073] It should be noted that the ring buffer stores 2.34s (i.e. 234 frames) of audio data, and returns 2.34s of audio data to the voice assistant application. However, when the voice assistant application performs voice wake-up, it only reads the first 2s (0 to 199 frames) of audio data for judgment. In addition, when the ring buffer stores audio data including the wake-up word, it does not specify the storage location of the wake-up word in the ring buffer.

[0074] In summary, the reason why the above-mentioned mobile phone fails to wake up the voice assistant application is that the mobile phone directly uses the original logic of transferring audio data, that is, directly uploads the audio data in a buffer containing the wake-up word to the voice assistant application. If the location of the wake-up word in the buffer is not within the first 2 seconds, for example, in the case where the wake-up word in the audio data of a buffer received by the voice assistant application is between the 150th frame and the 230th frame in the buffer, since part of the audio data corresponding to the wake-up word exceeds the range of the 0th frame to the 199th frame, the voice assistant application directly reads the first 2 seconds of audio data, which will result in the inability to read the wake-up word completely, and then cause the voice assistant application to wake up.

[0075] This embodiment can ensure that the voice assistant application reads the complete wake-up word when performing voice wake-up by selecting 2 seconds of audio data containing the wake-up word from the audio data in a buffer containing the wake-up word, splicing it to the position of the first 2 seconds of audio data in the buffer, and then uploading the audio data in the buffer to the voice assistant application. At this time, the mobile phone can successfully wake up the voice assistant application. Since a circular buffer stores 2.34 seconds of audio data, the location of the wake-up word at this time is at least 0.34 seconds away from the voice command stored in the next buffer.

[0076] Based on the aforementioned known judgment logic for one-word direct access, the reason why the above-mentioned mobile phone did not enter the one-word direct access after successfully waking up the voice assistant application is that there is a silent frame of at least 0.34 seconds between the location of the wake-up word and the voice command stored in the next buffer.

[0077] In order to solve the above problems, a voice wake-up method is provided in this embodiment. The mobile phone can select 2 seconds of audio data containing the wake-up word from the audio data in the buffer where the wake-up word is located. The mobile phone reads the 2 seconds of audio data containing the wake-up word when performing voice wake-up, and then reads the audio data in the subsequent buffer, so that after the mobile phone successfully wakes up the voice assistant application, it ensures that there is sound within 300ms after the wake-up word, thereby realizing the one-word direct access function. Among them, the mobile phone can select 2 seconds of audio data containing the wake-up word from the audio data in the buffer where the wake-up word is located, and splice it to the position of the first 2 seconds of audio data in the buffer, and read the first 2 seconds of audio data in the buffer when performing voice wake-up, so that the mobile phone can recognize the complete wake-up word when performing wake-up word recognition, and successfully wake up the voice assistant application.

[0078] First, this embodiment introduces the system architecture of the voice wake-up method. Figure 5 , Figure 5 A schematic diagram of the system architecture of a voice wake-up method provided in an embodiment of the present application. Figure 5 As shown in FIG. 1 , the system architecture of a mobile phone is divided into an AP module and an automatic data processing system (ADSP) module, or the system architecture of a mobile phone is divided into an AP module and an SCP module. Among them, on the MediaTek (MTK) platform, the system architecture of a mobile phone is divided into an AP module and an SCP module. On the Qualcomm (Qualcomm) platform, the system architecture of a mobile phone is divided into an AP module and an ADSP module. The ADSP module or the SCP module includes an audio processing module.

[0079] After the microphone (mic) of the mobile phone collects the user's audio data, it is uploaded to the codec (CodeC), and then uploaded to the audio processing module in the SCP module or ADSP module. The audio processing module then uploads the audio data to the AP module.

[0080] The AP module can be divided into the application layer, framework layer and HAL layer under the Android architecture. Among them, the application layer includes the voice assistant application, the framework layer includes the voiceprint recognition module, VAD module and ASR module. The hardware abstraction layer includes the audio (Aud io) HAL (abbreviated as AHAL) layer and the sound trigger (SoundT igger) HAL (abbreviated as STHAL) layer.

[0081] When the SCP module or the ADSP module detects a wake-up word, the detection event is uploaded to the AP module by the SCP module or the ADSP module, and then passed through the HAL layer and the framework layer, and is uploaded to the voice assistant application at the application layer.

[0082] After receiving the detection event, the voice assistant application can issue a streaming notification. The notification is sent to the HAL layer through the framework layer. The HAL layer can obtain the saved audio data from the ADSP module or the SCP module according to the identification stream included in the streaming notification.

[0083] The audio data then passes through the HAL layer and the framework layer to reach the voiceprint recognition module in the framework layer. After the voiceprint recognition module performs voiceprint recognition on the acquired audio data, it notifies the VAD module to detect the subsequent audio data, and then determines whether this voice wake-up is a normal wake-up or a direct one-word wake-up.

[0084] In this implementation, the mobile phone can shift and splice the audio data in the buffer where the wake-up word is located so that the 2 seconds of audio data including the wake-up word is located at the position of the first 2 seconds of audio data in the buffer.

[0085] In some embodiments, after detecting the wake-up word (first-level wake-up), the mobile phone records the time interval between this time point and the time when the AP side starts to obtain audio data, and then updates the frame header and frame tail information received when sending it to the voiceprint recognition module for second-level wake-up according to this time interval, so that the frame header and frame tail information obtained by the voiceprint recognition module during the second-level wake-up can correspond to the actual position of the wake-up word in the audio dump log.

[0086] Among them, after shifting and splicing the audio data in the buffer where the wake-up word is located, the frame header information and frame tail information of the data frame obtained by the voiceprint recognition module can be between the 110th frame and the 190th frame. Since there may be a delay in the process of the SCP module of the mobile phone sending the 2-second audio data including the wake-up word to the voiceprint recognition module, when the voiceprint recognition module recognizes the wake-up word, it will extend the frame header information in the obtained data frame forward by 5 frames and the frame tail information backward by 20 frames. If it exceeds the 199th frame, it will be extended to the 199th frame to ensure that the wake-up word is between the frame header and the frame tail.

[0087] It should be noted that the SCP module of the mobile phone will periodically detect the wake-up word. After detecting the wake-up word, the audio data in the buffer where the wake-up word is located will be shifted and spliced. Among them, the mobile phone will start voice wake-up only after detecting the wake-up word. Then, for a voice wake-up, the audio data including the wake-up word is in the front time period of the audio data received in this voice wake-up. Therefore, the mobile phone can use the circular buffer including the wake-up word as the first buffer.

[0088] In this embodiment, when the mobile phone writes the received audio data into the first buffer, the head pointer (head_ptr) remains unchanged at the position of the first buffer. The tail pointer (tail_ptr) is placed at the last audio data written into the first buffer. The length from the head pointer to the tail pointer is called the head size (head_size), and the length from the tail pointer to the end position of the first buffer is called the tail size (tail_size). In the circular buffer, the end position of the first buffer points to the position of the head pointer.

[0089] Among them, the mobile phone can set the first buffer to only store the wake-up word. After the mobile phone recognizes the wake-up word in the first buffer, it can no longer continue to store the voice information entered by the subsequent user.

[0090] It should be noted that when the first buffer writes audio data, it starts writing from the position of the tail pointer. Every time a frame of data is written, the tail pointer moves back one frame until the wake-up word is recognized and stops the tail pointer at the end position of the wake-up word.

[0091] It is understandable that if the ring buffer is full of 2.34s of audio data after the wake-up word is written, then the audio data from the position of the tail pointer onwards is the audio data first written into the ring buffer.

[0092] For example, after some environmental noise data is written into the first buffer, the wake-up word is written. After the wake-up word is written, the first buffer is full of 2.34s of audio data. Then, the audio data after the tail pointer is the environmental noise data.

[0093] Among them, if there is storage space left in the first buffer after the wake-up word is recognized, the mobile phone can make up the duration. Among them, the mobile phone can use silent frames to make up the duration of the audio data in the first buffer. Generally speaking, according to the user's speaking speed, the duration of the wake-up word is less than 1 second, then the audio data other than the wake-up word in the first buffer occupies at least 1.34 seconds. Then, since the tail pointer is at the end position of the wake-up word, the audio data that moves the tail pointer back for a set duration also does not contain the audio data of the wake-up word. The set duration is less than 1.34 seconds.

[0094] It should be noted that the position of the wake-up word in the first buffer is not fixed. For example, the wake-up word may be between the 0th frame and the 199th frame, or part of the wake-up word may be between the 200th frame and the 234th frame.

[0095] For scenarios where the wake-up word is at different positions in the first buffer, the shifting and splicing of the audio data in the first buffer may also be different. This embodiment can be divided into three situations to specifically introduce the shifting and splicing of the audio data in the first buffer.

[0096] In this implementation, the shifting and splicing of the audio data in the first buffer can be divided into three cases: the value of tai l_size is greater than or equal to 11520 bytes (corresponding to 0.34s), the value of tai l_size is less than 11520 bytes, and the value of tai l_size is equal to 11520 bytes.

[0097] Among them, assuming that the user says "Hello xx, open application 1", the mobile phone will store the above audio data through a circular buffer. The first buffer only stores the wake-up word "Hello xx". Assuming that after writing the wake-up word, the first buffer is full of audio data, "Open application 1" is stored in the second buffer. It can be understood that the duration corresponding to "Open application 1" is less than 2.34s, so it only occupies one circular buffer (second buffer). Then, the mobile phone will start to shift and splice the audio data in the first buffer.

[0098] See also Figure 6 , Figure 6 A schematic diagram of shifting and splicing audio data in a first buffer provided in an embodiment of the present application. Figure 6 As shown in (a), the tail pointer is placed at the end of the wake-up word, and the value of tai l_size is greater than 11520 bytes.

[0099] like Figure 6 As shown in (b), the mobile phone moves the tail pointer backward by 11520 bytes, and the values ​​of head_size and tail_size also change.

[0100] like Figure 6 As shown in (c), the mobile phone shifts the audio data corresponding to tai l_size to before the audio data corresponding to head_size, and then re-splices it with the audio data corresponding to head_size.

[0101] At this time, by Figure 6From (c), we can see that the wake-up word is located in the first 2 seconds of the first buffer. Then, the SCP module can send the audio data in the first buffer after the audio data is shifted and spliced ​​to the HAL layer, and then send it to the voiceprint recognition module of the framework layer. The voiceprint recognition module can perform voiceprint recognition on the first 2 seconds of the audio data in the first buffer after the shift and splicing processing. Since the complete wake-up word is recognized, the voice assistant application can be successfully awakened after the voiceprint recognition is successful.

[0102] It should be noted that, according to the logic of writing data to the first buffer, when writing data to the first buffer, after the tail pointer moves to the end (end position) of the first buffer, it starts to move backward from the position of the head pointer. Figure 6 In (a), we can only ensure that the tail pointer is at the end of the wake-up word, but we cannot assume that the wake-up word is completely at the end of the wake-up word. Figure 6 For example, if the wake-up word is written into the first buffer, the tail pointer is from Figure 6 If the movement starts within the last 0.34 seconds of the first buffer in (a), then part of the wake-up word will be located Figure 6 In the last 0.34 seconds of (a), the other part of the wake-up word is in Figure 6 The area corresponding to head_size in (a).

[0103] Therefore, the above-mentioned shifting and splicing of the audio data in the first buffer can ensure that all the wake-up words are in the first 2 seconds of the first buffer.

[0104] See also Figure 7 , Figure 7 A schematic diagram of shifting and splicing audio data in a first buffer provided in an embodiment of the present application. Figure 7 As shown in (a), the tail pointer is placed at the end of the wake-up word, and the value of tail_size is equal to 11520 bytes. Figure 7 As shown in (b), the mobile phone moves the tail pointer position directly backward by 11520 bytes, and the values ​​of head_size and tail_size also change. At this time, the tail pointer is at the end of the first buffer, so it can be considered that the value of head_size is the size of the entire buffer, and the value of tail_size is 0.

[0105] Then, the mobile phone shifts the audio data corresponding to tail_size to before the audio data corresponding to head_size, and then rejoins it with the audio data corresponding to head_size. In fact, the above does not move the position of the audio data corresponding to head_size in the first buffer, and the first buffer remains as Figure 7 as shown in (b).

[0106] At this time, the wake-up word is located in the first 2 seconds of the first buffer. Then, the SCP module can send the audio data of the first buffer after audio data shifting and splicing to the HAL layer, and then send it to the voiceprint recognition module of the framework layer.

[0107] It is understandable that if in the first buffer, the mobile phone detects that the wake-up word is exactly in the range of the 0th frame to the 199th frame, for example, when the value of the above-mentioned tai l_size is equal to 11520 bytes, the audio data in the first buffer does not need to be shifted and spliced ​​as described above. After the SCP module uploads the audio data of the first buffer to the AP module, the voiceprint recognition module can intercept the audio data of the previous 2 seconds for voiceprint recognition. Since the wake-up word is exactly in the range of the 0th frame to the 199th frame, the audio data of the previous 2 seconds intercepted by the voiceprint recognition module contains the wake-up word. Then, after the voiceprint recognition is successful, the mobile phone can wake up the voice assistant application.

[0108] See also Figure 8 , Figure 8 A schematic diagram of shifting and splicing audio data in a first buffer provided in an embodiment of the present application. Figure 8 As shown in (a), the tail pointer is placed at the end of the wake-up word, and the value of tail_size is less than 11520 bytes. Figure 8 As shown in (b), the mobile phone moves the tail pointer backward by 11520 bytes. Due to the characteristics of writing data in the circular buffer, after the tail pointer moves to the end of the first buffer, it moves backward from the head pointer until it moves 11520 bytes. At this time, the values ​​of head_size and tail_size also change.

[0109] like Figure 8 As shown in (c), the mobile phone shifts the audio data corresponding to tai l_size to before the audio data corresponding to head_size, and then re-splices it with the audio data corresponding to head_size.

[0110] At this time, by Figure 8 From (c), we can see that the wake-up word is located in the first 2 seconds of the first buffer. Then, the SCP module can send the audio data in the first buffer after the audio data splicing process to the HAL layer, and then send it to the voiceprint recognition module of the framework layer. The voiceprint recognition module can perform voiceprint recognition on the first 2 seconds of the audio data in the first buffer after the shift and splicing process. Since the complete wake-up word is recognized, the voice assistant application can be successfully awakened after the voiceprint recognition is successful.

[0111] See also Fig. 9 , Fig. 9 A schematic diagram of audio data of a ring buffer provided in an embodiment of the present application. The audio data in the first buffer is re-spliced ​​and directly spliced ​​with the audio data in the second buffer. The second buffer stores 2.34 seconds of audio data recorded after the wake-up word.

[0112] The second buffer is the voice command after the wake-up word is written from the first frame. If the interval between the wake-up word and the voice command entered by the user is too long, that is, there is no sound data in the first 300ms of the remaining audio, then even if the voice assistant application is successfully awakened, it will not enter the one-sentence direct access. In this embodiment, it is assumed that the user enters the wake-up word and voice command at a normal speed.

[0113] It can be understood that splicing the first buffer and the second buffer does not mean that there will be a step of splicing the buffers in the mobile phone, but is used to instruct the voiceprint recognition module of the mobile phone to read the audio data in each buffer in turn when reading audio data.

[0114] Fig. 9 The first buffer shown in (1a) is the first buffer after shifting and splicing when the value of tai l_size is greater than or equal to 11520 bytes.

[0115] like Fig. 9 As shown in (1b), the first buffer after the shifting and splicing process is directly spliced ​​with the second buffer.

[0116] Fig. 9 The first buffer shown in (2a) is the first buffer after shifting and splicing when the value of tai l_size is less than 11520 bytes.

[0117] like Fig. 9 As shown in (2b), the first buffer after the shifting and splicing process is directly spliced ​​with the second buffer.

[0118] It should be noted that the SCP module will only re-splice the audio data in the first buffer containing the wake-up word, and will not process the audio data in the buffer that does not contain the wake-up word, and will directly splice them in sequence according to the storage order.

[0119] Therefore, after the voiceprint recognition module receives the audio data of the first buffer under the above two splicing types, when it intercepts the audio data of the first 2 seconds for voiceprint recognition, the wake-up word must be within these 2 seconds of audio data, and the voice assistant application can be woken up after the voiceprint recognition is successful.

[0120] In this embodiment, after the above-mentioned shifting and splicing processing of the audio data in the first buffer, the SCP module can also select the audio data in the first buffer and upload only the audio data of the first 2s containing the wake-up word in the first buffer to the voiceprint recognition module before uploading the audio data in the first buffer to the voiceprint recognition module. In this way, the mobile phone can read only the audio data of the first 2s containing the wake-up word in the first buffer, and then read the subsequent audio data normally, so that after the mobile phone successfully wakes up the voice assistant application, it can detect that the subsequent 300ms of audio data has sound, thereby realizing the one-sentence direct access function.

[0121] See also Fig.10 , Fig.10 A schematic diagram of audio data of a ring buffer provided in an embodiment of the present application. When the value of tail_size is greater than or equal to 11520 bytes, the mobile phone moves the tail pointer in the first buffer back by 11520 bytes, and shifts and rejoins the audio data corresponding to the newly generated tail_size and the audio data corresponding to the head_size, and the first buffer is obtained as follows: Fig.10 At this time, for the first buffer, the SCP module only intercepts the first 2 seconds of audio data in the first buffer and uploads it to the HAL layer, and the last 340ms of audio data in the first buffer is no longer uploaded to the HAL layer.

[0122] Since the voice information after the wake-up word is stored in the ring buffer after the first buffer, the SCP module splices the contents of these buffers in order after the first buffer and uploads the audio data of the spliced ​​multiple buffers to the HAL layer. At this time, the HAL layer receives the audio clips such as Fig.10 (b) of FIG. 1 includes the first 2 seconds of audio data of the intercepted first buffer, and the buffer spliced ​​after the first buffer (such as Fig.10 The audio data of the second buffer (shown in (b)).

[0123] When the value of tail_size is less than 11520 bytes, the mobile phone moves the tail pointer in the first buffer back by 11520 bytes, and shifts and rejoins the audio data corresponding to the newly generated tail_size and the audio data corresponding to the head_size. The first buffer is as follows: Fig.10 At this time, for the first buffer, the SCP module only intercepts the first 2 seconds of audio data in the first buffer and uploads it to the HAL layer, and the last 340ms of audio data in the first buffer is no longer uploaded to the HAL layer.

[0124] Since the voice information after the wake-up word is stored in the ring buffer after the first buffer, the SCP module splices the contents of these buffers in order after the first buffer and uploads the audio data of the spliced ​​multiple buffers to the HAL layer. At this time, the HAL layer receives the audio clips such as Fig.10 As shown in (d): including the first 2 seconds of audio data of the intercepted first buffer, and the buffer spliced ​​after the first buffer (such as Fig.10 The audio data of the second buffer (shown in (d)).

[0125] Therefore, the voice assistant application of the mobile phone performs voiceprint recognition after receiving the first 2 seconds of audio data in the first buffer. After the voiceprint recognition is successful, the voice assistant application can be awakened because the complete wake-up word is recognized. Then, the mobile phone continues to recognize the audio data in the second buffer. Since the second buffer stores the voice instructions after the wake-up word entered by the user, the mobile phone can detect that there is sound in the subsequent 300ms, so the mobile phone can successfully turn on the one-sentence direct access function. At this time, the mobile phone can perform corresponding operations according to the recognized voice instructions.

[0126] In summary, after the one-sentence direct access function is turned on, the specific process of the mobile phone executing the "volume up" operation after the user says "Hello xx, turn up the volume" can be found in Fig.11 , Fig.11 A schematic diagram of a specific process for implementing one-word direct access provided in the embodiment of the present application. Fig.11 As shown, the SCP module of the kernel layer can periodically detect the wake-up word, which can also be called a hotword. Once the SCP module detects a hotword, it can report a detection event to the STHAL layer. Among them, the detection event is a first-level wake-up event. After the STHAL layer receives the detect event, it will be transparently transmitted to the voice assistant application through the framework layer.

[0127] Then, the voice assistant application sends the first stream (first identification stream) notification after receiving the detect event. The first identification stream can be called source 1999. After receiving the stream notification, the framework layer starts the stream of source 1999 through the get I nputFor function. Specifically, the framework layer initiates a stream notification to the AHAL layer, and the AHAL layer starts the stream of source 1999 by opening the input stream (open_input_stream).

[0128] The SCP module includes a memory block where the hotword is located, where the memory block may refer to a circular buffer, and the SCP module may store the audio data including the hotword through the first buffer. After the SCP module shifts and splices the audio data in the first buffer, it splices the first buffer that only retains the first 2s of audio data with the subsequent buffers and uploads them to the AP side.

[0129] Specifically, the audio data in the first buffer is transmitted through the AHAL layer and the framework layer and then sent to the voiceprint recognition module. The voiceprint recognition module will perform voiceprint recognition on the 2s audio data. If the voiceprint corresponding to the 2 seconds audio data is consistent with the voiceprint registered in the mobile phone, the voiceprint recognition is successful, otherwise the voiceprint recognition fails.

[0130] Since the audio data in the first buffer is shifted and spliced ​​as described above, the first 2 seconds of audio data includes a complete wake-up word. Then, after the voiceprint recognition is successful, the mobile phone can successfully wake up the voice assistant application.

[0131] Then, after the voiceprint recognition is successful, the voiceprint recognition module sends a notification to the VAD module, and the VAD module then determines whether there is a speaking sound in the first 300ms of the audio data in the second buffer after the first buffer. The VAD module can obtain the audio data in the circular buffer after the shift and splicing processing in the SCP module. If there is a speaking sound in the first 300ms, the voice assistant application recognizes that this is a direct wake-up.

[0132] Since the 2 seconds of audio data containing the wake-up word are adjacent to the second buffer, and the second buffer stores the audio data after the wake-up word, there is speech in the first 300ms. At this time, the VAD module detects that there is speech in the first 300ms and successfully enters the one-sentence direct wake-up process. At this time, the VAD module can send a notification message to the voice assistant application, which contains a message that this voice wake-up is a normal wake-up.

[0133] Then, the voice assistant application can recognize the voice command in the locally stored audio data through the ASR module and operate the mobile phone according to the command content (turn up the volume). Here, the locally stored audio data refers to the audio data received and stored in the buffer of the SCP module.

[0134] It is understandable that if the audio data stored in the first buffer includes the wake-up word and part of the voice command, the tail pointer will be placed at the tail of the last voice command written. After the above-mentioned shifting and splicing of the audio data in the first buffer, even if the first 2 seconds of audio data include a complete wake-up word, the voice assistant application can be successfully awakened, but when the mobile phone reads the audio data in the subsequent buffer, it will not be able to recognize the complete voice command, resulting in a failure of direct access. Therefore, the mobile phone can set the first buffer to only store the wake-up word to ensure that the subsequent buffer stores complete voice commands.

[0135] After shifting and splicing the audio data of the first buffer, the first buffer that intercepts the first 2 seconds of audio data is sequentially spliced ​​with the subsequent buffers. For a schematic diagram of the processed audio dump log, see Fig.12 From the above shifting and splicing of the audio data in the first buffer, it can be seen that the tail of the wake-up word will always be at the 2nd second position in the first buffer. Fig.12 As shown in the figure, there is no silent frame between the wake-up word ("Hello xx") and the voice command ("Turn up the volume"). At this point, the mobile phone can successfully turn on the one-sentence direct access function. After receiving the wake-up word and voice command entered by the user, the mobile phone can successfully wake up the voice assistant application and enter the one-sentence direct access function. It can directly perform the corresponding operation (i.e., turn up the volume) according to the recognized voice command.

[0136] In summary, after turning on the one-word direct access function, the mobile phone can only read the audio data of the first 2s containing the wake-up word in the first buffer, and then read the subsequent audio data normally, so that after successfully waking up the voice assistant application, the mobile phone can detect that the subsequent 300ms of audio data is audible, thereby realizing the one-word direct access function and directly executing the response operation according to the recognized voice command.

[0137] Exemplarily, the electronic device in the embodiment of the present application has an audio-related module, which may specifically include a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, or at least one of a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.

[0138] The following will describe the implementation of the embodiment of the present application in detail with reference to the accompanying drawings. The electronic device can be a mobile phone or a device. Taking the mobile phone as an example, the hardware structure of the electronic device 1400 is introduced. The hardware structure of the electronic device can refer to the detailed description of the electronic device 1400 in the embodiment of the present application, which will not be described in detail in the embodiment of the present application. Please refer to Fig.13 , Fig.13 A schematic diagram of the structure of an electronic device is shown. Fig.13 As shown, the electronic device 1400 may include: a processor 1410, an external memory interface 1420, an internal memory 1421, a mobile communication module 1450, a wireless communication module 1460, an audio module 1470, a speaker 1470A, a receiver 1470B, a microphone 1470C, an earphone interface 1470D, a sensor module 1480, a button 1490, a motor 1491, an indicator 1492, a display screen 1494, and a subscriber identification module (SIM) card interface 1495, a camera, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, and the like.

[0139] Among them, the above-mentioned sensor module 1480 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0140] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1400. In other embodiments of the present application, the electronic device 1400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0141] The processor 1410 may include one or more processing units, for example, the processor 1410 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0142] The processor 1410 can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. A memory can also be set in the processor 1410 for storing instructions and data. In some embodiments, the memory in the processor 1410 can be a cache memory. The memory can save instructions or data used by the processor 1410 or with a high frequency of use. If the processor 1410 needs to use the instruction or data, it can be directly called from the memory. Repeated access is avoided, the waiting time of the processor 1410 is reduced, and the efficiency of the system is improved.

[0143] In some embodiments, the processor 1410 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit audio (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 1410 may be connected to a touch sensor, an audio module, a wireless communication module, a display, a camera, and other modules through at least one of the above interfaces.

[0144] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 1400. In other embodiments of the present application, the electronic device 1400 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0145] The USB connector is an interface that complies with USB standard specifications and can be used to connect headphones to output audio stored in electronic devices through the headphones.

[0146] The electronic device 1400 can implement display functions through a GPU, a display screen 1494, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1410 may include one or more GPUs that execute program instructions to generate or change display information.

[0147] The display screen 1494 is used to display images, videos, etc. The display screen 1494 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini LED, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 1400 may include one or more display screens 1494.

[0148] In this embodiment, the display screen can be used to display various interfaces of the voice assistant application.

[0149] The external memory interface 1420 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1400. The external memory card communicates with the processor 1410 through the external memory interface 1420 to implement a data storage function. For example, files such as music and videos are stored in the external memory card. Or files such as music and videos are transferred from the electronic device to the external memory card.

[0150] The internal memory 1421 can be used to store computer executable program codes, which include instructions. The internal memory 1421 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 1400 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 1421 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (UFS), etc. The processor 1410 executes various functional methods or data processing of the electronic device 1400 by running instructions stored in the internal memory 1421 and / or instructions stored in a memory provided in the processor.

[0151] In this embodiment, the internal memory can store the voice information input by the user.

[0152] The electronic device 1400 can implement audio functions such as music playing and recording through the audio module 1470, the speaker 1470A, the receiver 1470B, the microphone 1470C, the headphone jack 1470D, and the application processor.

[0153] The audio module 1470 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 1470 can also be used to encode and decode audio signals. In some embodiments, the audio module 1470 can be arranged in the processor 1410, or some functional modules of the audio module 1470 can be arranged in the processor 1410.

[0154] The speaker 1470A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 1400 can listen to music through the speaker 1470A, or output an audio signal for a hands-free call. The receiver 1470B, also called a "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 1400 receives a call or a voice message, the voice can be received by placing the receiver 1470B close to the human ear.

[0155] Microphone 1470C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting their mouth close to the microphone 1470C to input the sound signal into the microphone 1470C. The electronic device 1400 can be provided with at least one microphone 1470C. In other embodiments, the electronic device 1400 can be provided with two microphones 1470C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 1400 can also be provided with three, four or more microphones 1470C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0156] In this embodiment, the mobile phone can collect audio data input by the user through a microphone.

[0157] The earphone interface 1470D is used to connect a wired earphone. The button 1490 may include a power button, a volume button, etc. The button 1490 may be a mechanical button. It may also be a touch button. The electronic device 1400 may receive a key input and generate a key signal input related to the user settings and function control of the electronic device 1400. The SIM card interface 1495 is used to connect a SIM card.

[0158] The methods in the following embodiments can all be implemented in the electronic device 1400 having the above hardware structure.

[0159] The present application also provides a chip system, such as Fig.14 As shown, the chip system 1500 includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 and the interface circuit 1502 can be interconnected via lines. For example, the interface circuit 1502 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 1502 can be used to send signals to other devices (such as processor 1501). Exemplarily, the interface circuit 1502 can read instructions stored in the memory and send the instructions to the processor 1501. When the instructions are executed by the processor 1501, the electronic device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0160] An embodiment of the present application also provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.

[0161] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0162] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0167] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A voice wake-up method, characterized in that: Applied to electronic devices, wherein the electronic devices are installed with a voice assistant application; The method comprises: The electronic device receives a voice information stream input by a user, the voice information stream is divided into a plurality of voice information of a first duration, and the plurality of voice information of the first duration include a first voice information including a wake-up word, and the wake-up word is used to wake up the voice assistant application; The electronic device performs wake-up word recognition on the voice information of the second duration in the first voice information; wherein the first duration is greater than the second duration, and the voice information of the second duration includes the wake-up word; In response to the electronic device recognizing that the wake-up word exists in the voice information of the second duration, the electronic device performs voice command recognition on the second voice information among the plurality of voice information of the first duration; the second voice information is the voice information input by the user after the first voice information in the voice information stream; In response to recognizing a voice instruction from the second voice information, the electronic device executes an operation corresponding to the voice instruction.

2. The method according to claim 1, characterized in that: The electronic device comprises a buffer, and the buffer is used to store the first voice information; The voice information of the second duration is voice information of a first preset number of frames starting from the first frame in the first voice information stored in the buffer, and the first preset number of frames corresponds to the second duration.

3. The method according to claim 2, characterized in that The electronic device does not recognize voice information other than the voice information of the second duration in the first voice information stored in the buffer.

4. The method according to claim 2 or 3, characterized in that: The electronic device includes a system control processor and a voiceprint recognition module; the electronic device performs wake-up word recognition on the voice information of the second duration in the first voice information, including: The system control processor of the electronic device intercepts the voice information of the second duration from the first voice information stored in the buffer; The system control processor of the electronic device sends the voice information of the second duration to the voiceprint recognition module; The voiceprint recognition module of the electronic device recognizes the wake-up word for the voice information of the second duration.

5. The method according to any one of claims 2 to 4, characterized in that: Before the electronic device performs voice command recognition on the second voice information among the plurality of voice information of the first duration, the method further includes: The electronic device detects that voice information of a second preset number of frames starting from the first frame in the second voice information contains sound data.

6. The method according to any one of claims 2 to 5, characterized in that: The buffer includes a head pointer and a tail pointer. Before the electronic device recognizes the wake-up word for the voice information of the second duration in the first voice information, the method further includes: The electronic device identifies the position of the wake-up word from the buffer and places the tail pointer at the tail of the wake-up word; The electronic device moves the position of the tail pointer by a third preset frame number; the third preset frame number is the frame number corresponding to the difference between the first duration and the second duration; The electronic device moves the voice information corresponding to the first area of ​​the buffer to the front of the voice information corresponding to the second area of ​​the buffer, and splices it with the voice information in the second area to obtain the spliced ​​first voice information; the first area is the area between the position of the tail pointer after the movement and the end position of the buffer, and the second area is the area between the position of the head pointer and the position of the tail pointer after the movement; the head pointer points to the starting position of the buffer, and the end position of the buffer points to the position of the head pointer.

7. The method according to claim 6, characterized in that The electronic device shifts the position of the tail pointer by a third preset frame number, including: The electronic device moves the tail pointer in a first direction, where the first direction is a direction pointing to an end position of the buffer zone; If the number of frames to be advanced is equal to the third preset number of frames before the tail pointer is advanced to the end position of the buffer, the advancement is terminated; If the number of frames moved is less than the third preset number of frames when the tail pointer moves to the end position of the buffer, the tail pointer continues to be moved in the first direction starting from the position of the head pointer until the number of frames moved is equal to the third preset number of frames.

8. An electronic device, characterized in that: The electronic device comprises: a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code comprises computer instructions, and when the computer instructions are executed by the electronic device, the electronic device executes the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed in an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed in an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Voice processing method and equipment

    CN109994106A

  • Voice data processing method, device and system and storage medium

    CN110992953A

  • Voice control method and device, electronic equipment and storage medium

    CN112382285A

  • Speech recognition method and device, storage medium and electronic device

    CN115881114A

  • Speech recognition method and device, equipment and storage medium

    CN116895277A