A voiceprint wake-up method and system

Abstract

This application provides a voiceprint wake-up method and system. By integrating the two core functions of speaker confirmation and wake-up word confirmation end-to-end, it eliminates the need to segment the wake-up word before feeding it into the voiceprint model, thus avoiding inaccurate voiceprint extraction caused by inaccurate wake-up word segmentation. In this method, voiceprint confirmation and wake-up word confirmation can be performed simultaneously, avoiding the time delay of two-stage voiceprint wake-up. Furthermore, this method uses idempotent encoding for all possible scenarios of wake-up word and speaker confirmation, allowing the model to directly output the status of voiceprint confirmation and speaker verification without adjusting thresholds to confirm whether a wake-up word or target speaker has been detected. This eliminates the need for manual threshold adjustment during deployment, thereby reducing deployment complexity.

Description

Technical Field

[0001] This invention relates to voiceprint wake-up technology, and more particularly to a voiceprint wake-up method and system. Background Technology

[0002] Voiceprint wake-up, also known as speaker-specific wake-up, adds a speaker verification task to voice wake-up. This allows voice assistants to not only respond after the user speaks the wake word but also distinguish the speaker's identity, enabling them to adopt different human-computer interaction strategies for different speakers. Therefore, it is widely used in voice interaction systems such as mobile terminals, smart homes, in-vehicle navigation, and smart wearables. More specifically, the voiceprint wake-up task continuously receives voice messages in the background and automatically detects predefined wake words. Once a corresponding wake word is detected, it needs to verify whether the wake word was spoken by a specific speaker. This ensures that only the specific speaker can speak the wake word to wake up the device and put it into a specific working state.

[0003] Traditional voiceprint wake-up is a two-stage process: a voiceprint model is cascaded after the wake-up model. After the wake-up model is activated, the speaker's speech fragment is sent to the voiceprint model for identity verification. The specific process is as follows: Figure 1 As shown: The wake word model is used to detect whether an audio sequence contains a wake word. Once a wake word is detected, the wake word boundary is confirmed using posterior probability and the audio is cut out. Then, the cut-out wake word is sent to the voiceprint model to extract the wake word voiceprint (generally, the voiceprint model here is content-related). Finally, the cosine similarity between the wake word voiceprint and the target speaker's voiceprint is calculated. When the cosine similarity is greater than the threshold, it is confirmed that the target speaker said the wake word.

[0004] The aforementioned traditional voiceprint wake-up method has four drawbacks:

[0005] 1. Difficulty in Completely Extracting Wake-up Words: The voiceprint models used in voiceprint wake-up systems are generally content-dependent, meaning they can only recognize the voiceprint corresponding to the wake-up word. If the input speech is not the wake-up word or is an incomplete wake-up word, the voiceprint performance will drop sharply. The posterior probability output by the wake-up model cannot accurately obtain the temporal boundaries of the wake-up word fragment, thus preventing the accurate and complete input of the wake-up word fragment into the voiceprint model. This significantly affects the accuracy of speaker confirmation.

[0006] 2. High latency: The voiceprint model in the second stage only starts working after the wake-up model is activated, which further increases the latency of the voiceprint wake-up system. If the latency is too high, it will greatly affect the user experience. Therefore, the wake-up model in the two stages must be balanced between computational load and performance: the larger the voiceprint model size, the better the performance, but this will increase the system latency; the smaller the voiceprint model, the lower the latency, but the poorer the environmental robustness.

[0007] 3. High Deployment Difficulty: The output of the voiceprint model is a high-dimensional vector (voiceprint). It determines whether the speaker is the same person by calculating the cosine similarity between the target speaker's voiceprint and the voiceprint extracted from the speech segment. Therefore, a reasonable threshold needs to be set for the cosine similarity to determine whether two voiceprints belong to the same person. If the threshold is too high, it will lead to a high false rejection rate; if the threshold is too low, it will lead to a high false acceptance rate. Furthermore, the reasonable threshold varies greatly in different use cases, which significantly increases the deployment difficulty of voiceprint wake-up.

[0008] 4. In a voiceprint wake-up system, the voiceprint model is related to the wake-up word. The content representation of the wake-up word can be used to help the voiceprint model generate a voiceprint representation related to the wake-up word. Obviously, in a two-stage voiceprint wake-up system, the wake-up model and the voiceprint model are isolated and cascaded, making it impossible to integrate the content representation into the voiceprint representation. Summary of the Invention

[0009] The purpose of this invention is to provide a voiceprint wake-up method and system, which integrates the two core functions of speaker confirmation and wake-up word confirmation end-to-end, and projects the output as idempotent set encoding, thereby making voiceprint wake-up have lower latency, higher accuracy and lower deployment difficulty.

[0010] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0011] According to a first aspect of the present invention, a voiceprint wake-up method is provided, comprising the following steps:

[0012] S01, Receive voice sequence x;

[0013] S02, extract the acoustic features fea of ​​the speech sequence x;

[0014] S03, the acoustic feature fea is fed into the voiceprint model to extract the frame-level speaker representation emb;

[0015] S04, Calculate the frame-level speaker representation emb and the pre-stored target speaker voiceprint emb. tar The cosine similarity between them is Sim;

[0016] S05, the cosine similarity Sim and the acoustic representation fea are concatenated to obtain Rep;

[0017] S06, feed Rep into the wake word detection model and output a high-dimensional vector bott_neck that contains speaker representation and wake word representation;

[0018] S07, use a linear layer (Linear) to project bot_neck onto a power set encoding table and output the voiceprint wake-up state logit. Each voiceprint wake-up state in the power set encoding table has a unique number, and each number corresponds to an output node of the linear layer (Linear).

[0019] S08, normalize logit to the probability distribution law prob with a sum of 1. The probability distribution law prob represents the posterior probability of the voiceprint wake-up state. The node number with the largest posterior probability is used as the value of the voiceprint wake-up state state of the current frame.

[0020] In one embodiment, the power set encoding table includes:

[0021] The first number corresponds to the case where the speech sequence contains a wake word and the speaker is the target speaker;

[0022] The second number corresponds to the case where the speech sequence contains a wake word and the speaker is not the target speaker;

[0023] The third number corresponds to the case where the speech sequence does not contain a wake word and the speaker is the target speaker;

[0024] The fourth number corresponds to the case where the speech sequence does not contain a wake word and the speaker is not the target speaker.

[0025] In one embodiment, step S08 further includes: if the value of the voiceprint wake-up state of the current frame is a first number, then it is considered that the target speaker is saying the wake-up word, and a wake-up operation is performed.

[0026] In one embodiment, the voiceprint model uses the RepVGG network as the backbone network, employs a multi-branch structure during training, and merges the branches during inference.

[0027] In one embodiment, each convolutional layer of the RepVGG network used in the voiceprint model employs causal convolution.

[0028] In one embodiment, the voiceprint model is initialized using a pre-trained model and then participates in the backpropagation of the voiceprint wake-up network and is updated.

[0029] In one embodiment, the wake word detection model uses a temporal convolutional network as its backbone network.

[0030] In one embodiment, the temporal convolutional network employs causal convolution.

[0031] According to a second aspect of the present invention, a voiceprint wake-up method is provided, comprising the following steps:

[0032] Input the audio sequence to be detected;

[0033] Calculate the timbre similarity between the audio sequence to be detected and the pre-stored target speaker's voice;

[0034] The timbre similarity information and the acoustic features of the audio sequence to be detected are concatenated and input into the wake word detection model, and the output is the power set encoding corresponding to all possible situations of the voiceprint wake-up state;

[0035] Whether to wake up is determined based on the posterior probability of the corresponding power set encoding for each situation.

[0036] According to a third aspect of the present invention, a voiceprint wake-up system is provided, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement the voiceprint wake-up method as described in the first or second aspect when the computer program is executed.

[0037] The beneficial effects of the embodiments of the present invention are:

[0038] Unlike traditional two-stage voiceprint wake-up, this invention integrates the two core functions of speaker verification and wake-up word verification end-to-end. It only requires the input of the target speaker's voiceprint as a reference for speaker verification and an input audio sequence for wake-up word detection. Its output is all possible combinations of voiceprint wake-up. Since the model can directly output the status of voiceprint verification and speaker validation, there is no need to adjust thresholds to confirm whether a wake-up word or target speaker has been detected. This eliminates the need for manual threshold adjustment during deployment, thus reducing deployment complexity.

[0039] Compared to the two-stage voiceprint wake-up model, which first extracts the wake word segment to obtain the voiceprint and then calculates the voiceprint cosine distance, this invention integrates the voiceprint and wake-up module end-to-end. It eliminates the need to extract the wake word before feeding it into the voiceprint model, thus avoiding inaccurate voiceprint extraction caused by inaccurate wake word segmentation. Furthermore, this invention allows for simultaneous voiceprint confirmation and wake word confirmation, eliminating the need to confirm the wake word first and then the voiceprint, thereby avoiding the time delay of the two-stage voiceprint wake-up method. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0042] Figure 1 This is a traditional two-stage voiceprint wake-up flowchart;

[0043] Figure 2 This is an overall flowchart of the voiceprint wake-up method provided in the embodiments of this application;

[0044] Figure 3 This is a flowchart illustrating the voiceprint wake-up method provided in the embodiments of this application.

[0045] Figure 4 This is a model network structure diagram of the voiceprint wake-up method provided in the embodiments of this application. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0047] like Figure 2 As shown, this embodiment provides an end-to-end voiceprint wake-up method. It only requires inputting the target speaker's voiceprint as a reference for speaker confirmation and an input audio sequence for wake-up word detection. Its output is all possible combinations of voiceprint wake-up states. In this application, all possible voiceprint wake-up state combinations are encoded into unique numbers, and these numbers are mapped one-to-one with the output nodes of the voiceprint wake-up network. This process is called power set encoding, which will be described in detail below.

[0048] like Figure 3 As shown, the specific process of this embodiment mainly includes a voiceprint comparison branch (S03-S04) and a wake-word confirmation branch (S05-S07). The voiceprint comparison branch is responsible for comparing the target speaker's voiceprint with the speaker representation extracted from the current speech, thereby obtaining the similarity information between the speech sequence and the target speaker. The input to the wake-word confirmation branch is the timbre similarity information in step S04 and the acoustic representation of the speech sequence in step S02. The detection model determines whether the wake-word was spoken by the target speaker from the acoustic representation and the speaker similarity representation.

[0049] The following is a detailed description of each branch of the present invention in sequence:

[0050] Steps S01 and S02 are used to extract acoustic representations from the speech sequence. Specifically, step S01 receives the speech sequence to be tested, x. Step S02 extracts the fbank features fea∈R of the speech sequence x. T×D Where T is the number of frames and D is the dimension of the fbank.

[0051] The voiceprint comparison branch is S03->S04. Specifically, in step S03, the fbank features are fed into the voiceprint model SpkrEnc to extract the frame-level speaker representation emb∈R. T×D′ (D′ is the dimension of representation):

[0052] emb = SpkrEnc(fea)

[0053] To reduce the computational complexity of the voiceprint model, this embodiment uses RepVGG as the backbone network, thereby reducing system power consumption and latency. Furthermore, a multi-branch structure is employed during training to retain more nonlinear representations, and branch merging is used during inference to reduce the network's computational load.

[0054] Furthermore, to reduce inference latency, each convolutional layer of RepVGG used in this embodiment employs causal convolution, which means that zeros are padded to the left of each convolutional layer by (k-1)×d (where k is the kernel size and d is the convolutional dilation). In addition, the voiceprint model SpkrEnc is initialized using a pre-trained model and then participates in the backpropagation and updates of the voiceprint wake-up network.

[0055] Step S04: Calculate the target speaker's voiceprint emb. tar The cosine similarity Sim∈R between the frame-level speaker representation emb and the speaker representation at the frame level. T :

[0056] Sim= <emb,emb tar >

[0057] This similarity score represents the timbre similarity between the speaker in the speech sequence and the target speaker; a higher cosine similarity value indicates a higher similarity. The subsequent wake-word confirmation branch uses this similarity score to determine whether the speech sequence contains a wake-word spoken by the target speaker.

[0058] The target speaker's voiceprint emb tar The voiceprint is pre-saved after the target speaker registers. When the voiceprint wake-up system runs, it uses this saved voiceprint for timbre similarity comparison. It is only activated when the system detects that the target speaker has uttered the wake word. The registration process involves extracting the target speaker's voiceprint using a voiceprint model, and its workflow is the same as traditional voiceprint extraction steps: extracting frame-level speaker representations using the voiceprint model and then temporally pooling these representations to obtain the voiceprint. This will not be elaborated further here.

[0059] Steps S05->S06->S07 constitute the wake-word confirmation branch. Specifically, in step S05, the cosine similarity Sim is concatenated with the acoustic representation fea to obtain Rep∈R. T×(D+1) The data is then fed into the wake word detection model KWsDetec in step S6 to obtain the output bottom_neck∈R. T×D″ (D″ is the dimension of the feature):

[0060] Rep = cat(fea, Sim)

[0061] bott_neck = KwsDetec(Rep)

[0062] The wake-word detection model KwsDetec is used to detect whether the target speaker utters a wake-word in the input representation. Therefore, `bott_neck` is a high-dimensional vector containing both speaker representation and wake-word representation. Considering that the SpkrEnc voiceprint model backbone network is wake-word related and can provide both timbre and wake-word representation, this embodiment uses a lighter-weight temporal convolutional network as the backbone network of the KwsDetec wake-word detection model, compared to the traditional two-stage voiceprint wake-up system. Similar to the voiceprint model, the convolutional layers in the temporal convolutional network also employ causal convolution to reduce model latency.

[0063] Traditional two-stage voiceprint wake-up requires manual verification of the cosine distance threshold between the voiceprint of the input audio and the target speaker's voiceprint, depending on the specific usage scenario. This increases the deployment difficulty of the voiceprint wake-up system. Furthermore, the two-stage modeling approach ignores the correlation between the wake-up word and the voiceprint. To address these issues, step 07 of this invention projects the bottleneck_neck feature onto all possible voiceprint wake-up states logit∈R. T×4 Each combination of voiceprint wake-up is assigned a unique number, which corresponds to an output node of the linear layer (this encoding method is referred to as power set encoding in this application):

[0064] logit = Linear(bott_neck)

[0065] In this embodiment, the power set encoding is shown in the table below:

[0066] The voice sequence contains a wake word The speaker is the target speaker. Power set number √ √ 0 √ x 1 x √ 2 x x 3

[0067] Among them, power set number 0 corresponds to the state where the target speaker says the wake word; power set number 1 corresponds to the state where the target speaker does not say the wake word; power set number 2 corresponds to the state where the non-target speaker says the wake word; and power set number 3 corresponds to the state where the non-target speaker does not say the wake word.

[0068] Therefore, the two single-label classification problems in the two-stage voiceprint wake-up are transformed into a multi-label classification problem. During engineering deployment, the logit output of the linear layer is used to explicitly represent whether the wake-up word and the target speaker have been detected. There is no need to adjust the threshold of the wake-up or voiceprint module, which greatly reduces the deployment difficulty.

[0069] In step S08, softmax is used to normalize the output logit from step S07 to the probability distribution prob∈R where the sum is 1. T×4 This distribution law represents the posterior probability of voiceprint-activated state:

[0070] prob = softmax(logit)

[0071] The node with the highest posterior probability represents the voiceprint wake-up state of the current frame:

[0072] state=argmax(prob),state∈R T

[0073] In this embodiment, once the value of state is 0, it indicates that the system has detected the target speaker saying the wake word, and can then perform subsequent wake-up operations.

[0074] Figure 4 The diagram shows the network structure of the voiceprint wake-up model used in this application, from top to bottom corresponding to the voiceprint comparison branch and the wake-up word confirmation branch. The voiceprint comparison branch uses RepVGG to extract frame-level speaker representations from the fbank features and calculates the similarity between the representations and the target speaker's voiceprint. The wake-up word confirmation branch concatenates the cosine similarity with the fbank features and feeds it into a temporal convolutional network to extract bottleneck features containing both speaker and wake-up word representations. Then, a linear layer projects the bottleneck features onto the voiceprint wake-up state, and finally, softmax is used to obtain the posterior probability of each state.

[0075] It is readily understood that embodiments of this application also provide a voiceprint wake-up system, which includes at least a memory and a processor.

[0076] The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the aforementioned voiceprint wake-up method; the data storage area may store data involved in the aforementioned voiceprint wake-up method.

[0077] A processor may include one or more processing cores. The processor executes instructions, programs, code sets, or instruction sets stored in memory, and calls data stored in memory to perform various functions and process data as described in this application. The processor may be at least one of a specific application-specific integrated circuit, a digital signal processor, a digital signal processing device, a programmable logic device, a field-programmable gate array, a central processing unit, a controller, a microcontroller, and a microprocessor. It is understood that, for different devices, the electronic devices used to implement the above-described processor functions may also be other types, and the embodiments of this application do not specifically limit the specific implementation.

[0078] If the voiceprint wake-up method described in the embodiments of this application is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of the present invention are not limited to any specific hardware and software combination.

[0079] Accordingly, embodiments of the present invention also provide a computer storage medium storing computer-executable instructions, which, when executed by a processor, implement the voiceprint wake-up method described above in the embodiments of the present invention.

[0080] In summary, the embodiments of this application provide an end-to-end, low-latency, and easily deployable voiceprint wake-up method and system. Compared with the two-segment voiceprint wake-up model, which first extracts the wake word fragment to extract the voiceprint and then calculates the voiceprint cosine distance, this embodiment integrates the voiceprint and the wake-up module end-to-end, eliminating the need to extract the wake word and then send it into the voiceprint model. This avoids inaccurate voiceprint extraction caused by inaccurate wake word segmentation.

[0081] Furthermore, in this embodiment, voiceprint verification and wake-up word verification can be performed simultaneously, eliminating the need to verify the wake-up word first and then the voiceprint, thus avoiding the time delay of two-stage voiceprint wake-up. To further reduce system power consumption and latency, this invention uses RepVGG as the voiceprint backbone network, thereby retaining more nonlinear representations during training and reducing network computation through reparameterization during inference. Considering that the voiceprint backbone network is wake-up word-dependent and can provide both timbre and wake-up word representations, unlike traditional two-stage voiceprint wake-up systems, this embodiment uses a lighter-weight temporal convolutional network as the wake-up word detection backbone network. To address the convolution latency issue, both the voiceprint and wake-up backbone networks in this invention use causal convolution. Through the above model structure design and selection, the voiceprint wake-up of this invention is compact and has low latency.

[0082] Traditional two-stage voiceprint wake-up models, whether using the wake-up model or the voiceprint model, require adjusting appropriate thresholds during deployment to achieve suitable false rejection and false acceptance rates. However, finding the right threshold often requires manual adjustment using a large development set based on different use cases, which significantly increases the deployment difficulty of the voiceprint wake-up system. To address this issue, this embodiment uses idempotent encoding for all possible scenarios of the wake-up word and speaker confirmation. Therefore, the model can directly output the status of voiceprint confirmation and speaker verification without adjusting thresholds to confirm whether the wake-up word or target speaker has been detected. This eliminates the need for manual threshold adjustment during deployment, thereby reducing deployment difficulty.

[0083] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0084] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0085] The above description is merely a preferred example of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A voiceprint wake-up method, characterized in that, Includes the following steps: S01, Receive voice sequence x; S02, Extract the acoustic features of speech sequence x ; S03, acoustic features Input into the voiceprint model and extract the speaker representation at the frame level. ; S04, Calculate frame-level speaker representation Compared with the pre-stored target speaker's voiceprint Cosine similarity between ; S05, cosine similarity Acoustic characteristics After splicing, the result is ; S06, The data is fed into a wake word detection model, which outputs a high-dimensional vector containing speaker representations and wake word representations. ; S07, using a linear layer Will Projecting the signal onto the power set encoding table, the voiceprint wake-up status is output. In the power set coding table, each voiceprint wake-up state has a unique number, and each number corresponds to a linear layer. One of the output nodes; S08, will Probability distribution law normalized to a sum of 1 Probability distribution law The posterior probability representing the voiceprint wake-up state is used, and the node number with the highest posterior probability is taken as the voiceprint wake-up state of the current frame. The value of .

2. The voiceprint wake-up method according to claim 1, characterized in that, The power set encoding table includes: The first number corresponds to a speech sequence that contains a wake word and the speaker is the target speaker. The second number corresponds to the state where the voice sequence contains a wake word and the speaker is not the target speaker; The third number corresponds to a speech sequence that does not contain a wake word and the speaker is the target speaker; The fourth number corresponds to a speech sequence that does not contain a wake word and the speaker is not the target speaker.

3. The voiceprint wake-up method according to claim 2, characterized in that, Step S08 further includes: if the current frame is in voiceprint wake-up state If the value is the first number, it is considered that the target speaker is saying the wake word, and the wake-up operation is executed.

4. The voiceprint wake-up method according to claim 1, characterized in that, The voiceprint model uses the RepVGG network as its backbone network, employs a multi-branch structure during training, and merges the branches during inference.

5. The voiceprint wake-up method according to claim 4, characterized in that, The RepVGG network used in the voiceprint model employs causal convolution in each convolutional layer.

6. The voiceprint wake-up method according to claim 5, characterized in that, The voiceprint model is initialized using a pre-trained model, and then participates in the backpropagation of the voiceprint wake-up network for updating.

7. The voiceprint wake-up method according to claim 1, characterized in that, The wake word detection model uses a temporal convolutional network as its backbone network.

8. The voiceprint wake-up method according to claim 7, characterized in that, The temporal convolutional network employs causal convolution.

9. A voiceprint wake-up system, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the voiceprint wake-up method as described in any one of claims 1 to 8 when executing the computer program.

Images