Fine-tuning method and device of wake-up word and storage medium

Abstract

The application provides a fine-tuning method and device of a wake-up word and a storage medium, the method comprising: inputting a sample voice signal input by a user as a wake-up word into a voice feature network to extract first sample wake-up voice features; if the sample voice signal is not a dialect, repeatedly monitoring a wake-up event of the user; if the first reference voice signal is a dialect, separating accent features when the user speaks the dialect from the first reference wake-up voice features according to the first sample wake-up voice features; integrating the accent features into the first sample wake-up voice features according to the voice feature network to obtain second sample wake-up voice features; and performing a wake-up operation according to a target voice signal, the first sample wake-up voice features and the second sample wake-up voice features when the target voice signal is received. The embodiment makes the wake-up word adapt to the accent change when the user speaks the dialect, thereby improving the success rate of using the wake-up word to wake up the device.

Description

Technical Field

[0001] This invention belongs to the technical field of speech processing, and in particular relates to a method, device and storage medium for fine-tuning wake words. Background Technology

[0002] Intelligent electronic devices such as speakers, televisions, and lamps provide personalized services to users, usually relying on the user's personalized wake word. That is, after the electronic device detects the wake word spoken by the user, it enters the working state from the standby state and loads the various parameters configured for that user.

[0003] Currently, wake words are usually recorded when activating or resetting electronic devices for the first time. Users typically pronounce the wake word in a standardized language such as Mandarin or English for the electronic device to record. However, when users use smart devices, they are often in a relaxed state, and the wake words they pronounce may have a certain accent, which leads to a decrease in the success rate of waking up the device using a wake word. Summary of the Invention

[0004] In view of this, the present invention provides a method, device and storage medium for fine-tuning a wake word, in order to improve the success rate of waking up a device using a wake word.

[0005] A first aspect of the present invention provides a method for fine-tuning a wake word, comprising:

[0006] When a sample voice signal is received as a wake-up word by the user, the sample voice signal is input into the voice feature network to extract the first sample wake-up voice feature.

[0007] If the sample speech signal is not a dialect, then the user is monitored for repeated wake-up events; the repeated wake-up event occurs when the wake-up operation succeeds after the wake-up operation fails.

[0008] The first reference wake-up speech feature extracted from the speech feature network when the first reference speech signal fails to query the wake-up operation.

[0009] If the first reference speech signal is a dialect, then the accent features of the user speaking the dialect are separated from the first reference wake-up speech features based on the first sample wake-up speech features.

[0010] The accent features are integrated into the first sample wake-up speech features based on the speech feature network to obtain the second sample wake-up speech features;

[0011] Upon receiving the target voice signal, a wake-up operation is performed based on the target voice signal, the first sample wake-up voice feature, and the second sample wake-up voice feature.

[0012] A second aspect of the present invention provides a device for fine-tuning a wake word, comprising:

[0013] The first sample feature extraction module is used to input the sample voice signal into the voice feature network to extract the first sample wake-up voice features when it receives a sample voice signal entered by the user as a wake-up word.

[0014] A repeated wake-up event monitoring module is used to monitor repeated wake-up events for the user if the sample voice signal is not a dialect; the repeated wake-up event occurs when the wake-up operation succeeds after the wake-up operation fails.

[0015] The reference feature query module is used to query the first reference wake-up speech feature extracted from the speech feature network when the wake-up operation fails.

[0016] An accent feature separation module is used to separate the accent features of the user speaking a dialect from the first reference wake-up speech features based on the first sample wake-up speech features if the first reference speech signal is a dialect.

[0017] The second sample feature generation module is used to integrate the accent features into the first sample wake-up speech features based on the speech feature network to obtain the second sample wake-up speech features.

[0018] The wake-up operation execution module is used to perform a wake-up operation based on the target voice signal, the first sample wake-up voice feature, and the second sample wake-up voice feature when a target voice signal is received.

[0019] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the wake word fine-tuning method as described in the first aspect above.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the wake word fine-tuning method as described in the first aspect above.

[0021] A fifth aspect of the present invention provides a computer program product that, when run on a computer, causes the computer to perform the wake word fine-tuning method as described in the first aspect above.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0023] In this embodiment, upon receiving a sample voice signal entered by the user as a wake-up word, the sample voice signal is input into a voice feature network to extract the first sample wake-up voice feature. If the sample voice signal is not a dialect, the user is monitored for repeated wake-up events. A repeated wake-up event occurs when a wake-up operation succeeds after a previous failure. The first reference wake-up voice feature extracted from the first reference voice signal at the time of the failed wake-up operation is queried. If the first reference voice signal is a dialect, the user's accent features when speaking the dialect are separated from the first reference wake-up voice feature based on the first sample wake-up voice feature. The accent features are then integrated into the first sample wake-up voice feature based on the voice feature network to obtain the second sample wake-up voice feature. Upon receiving a target voice signal, a wake-up operation is performed based on the target voice signal, the first sample wake-up voice feature, and the second sample wake-up voice feature. This embodiment classifies the accent features of dialects from non-dialect wake-up voices, thereby fine-tuning the voice features of the wake-up word online. While maintaining a personalized wake-up word, the wake-up word adapts to the user's accent changes when speaking a dialect, thus improving the success rate of waking up the device using the wake-up word.

[0024] Furthermore, this embodiment is relatively simple to operate, suitable for on-device computation, and can ensure user privacy and security. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of a method for fine-tuning a wake word provided in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of a repeated wake-up event provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram illustrating the integration of accent features according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of a wake-up word fine-tuning device provided in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the present invention. However, those skilled in the art will recognize that the present application may be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted to avoid unnecessary detail that could obscure the description of the present application.

[0032] The technical solution of the present invention will be illustrated below through specific embodiments.

[0033] Reference Figure 1 The diagram illustrates a method for fine-tuning a wake word according to an embodiment of the present invention, which may specifically include the following steps:

[0034] Step 101: When a sample voice signal is received as a wake-up word by the user, the sample voice signal is input into the voice feature network to extract the first sample wake-up voice feature.

[0035] When a user enters a new wake word for an electronic device, the system can receive a sample voice signal spoken by the user. The sample voice signal is then input into a preset voice feature network to extract features containing the user's voiceprint and the wake word content, which is recorded as the first sample wake-up voice feature.

[0036] For example, speech feature networks include X-vector, WeSpeaker ResNet34, Multi-TaskTransformer KWS, Transformer and its improved structures (such as Wav2Vec2 Base, HuberT Base), etc.

[0037] Step 102: If the sample speech signal is not a dialect, then listen for repeated wake-up events for the user.

[0038] In this embodiment, the wake-up voice features of the first sample can be input into a preset language classifier for binary classification, thereby identifying the language type of the sample voice signal.

[0039] For example, language classifiers include SVM (Support Vector Machine), TDNN (Time Delay Neural Network), SLNet (Single Task Dialect Recognition Network), and so on.

[0040] If the language type of the sample speech signal is not a dialect, the user may use standardized language to pronounce the wake word precisely. In this case, during the subsequent use of smart home appliances, the system will monitor the user for repeated wake-up events.

[0041] Among them, the repeated wake-up event is two consecutive wake-up operations, in which the wake-up operation fails and then succeeds. The first wake-up operation may be that the user speaks the wake-up word in dialect, causing the first wake-up operation to fail. The user may realize the influence of dialect. The second wake-up operation may be that the user speaks the wake-up word in standardized language, causing the second wake-up operation to succeed.

[0042] In specific implementations, such as Figure 2 As shown, the electronic device continuously listens to voice signals in standby mode. When it receives the first reference voice signal (i.e. the latest voice signal), it inputs the first reference voice signal into the voice feature network to extract features containing the user's voiceprint and voice content, which are denoted as the first reference wake-up voice features.

[0043] Calculate the similarity (e.g., cosine similarity) between the first reference wake-up speech features and the first sample wake-up speech features.

[0044] If the similarity between the first reference wake-up voice feature and the first sample wake-up voice feature is greater than or equal to the first confidence level and less than the second confidence level, then the wake-up operation is determined to have failed, the wake-up failure event Event1 is recorded, and at the same time, a timer is started (the timer duration is T).

[0045] If no voice signal is received when the start timer ends, the timer is released.

[0046] When the second reference voice signal (i.e. the latest voice signal) is received before the start timer expires, the second reference voice signal is input into the voice feature network to extract features containing the user's voiceprint and voice content, which are denoted as the second reference wake-up voice features.

[0047] Calculate the similarity (e.g., cosine similarity) between the second reference wake-up speech features and the first sample wake-up speech features.

[0048] If the similarity between the second reference wake-up voice feature and the first sample wake-up voice feature is greater than or equal to the second confidence level, then the wake-up operation is performed, i.e. the wake-up operation is successful, the wake-up success event Event2 is recorded, the electronic device is switched from the standby state to the working state, and, in response to the occurrence of Event1 and Event2 within T, it is determined that a repeated wake-up event has occurred.

[0049] Step 103: Query the first reference wake-up speech features extracted from the first reference speech signal when the wake-up operation fails in the speech feature network.

[0050] When the number of repeated wake-up events exceeds a preset update threshold, the first reference wake-up speech feature can be extracted from the first reference speech signal when the wake-up operation fails and extracted in the speech feature network.

[0051] Step 104: If the first reference speech signal is a dialect, then the accent features of the user speaking the dialect are separated from the first reference wake-up speech features based on the first sample wake-up speech features.

[0052] In practical applications, the feature space layout of a speech feature network satisfies the linear assumption and can be linearly decomposed. Therefore, the wake-up speech feature p of the first sample can be expressed as p = c. kw +n p , where c kw For the content features of the wake word, n p For non-dialectal random noise, the first reference wake-up speech features can be d kw Represented as d kw =c kw +a dialect +n d , where a dialect As a characteristic of dialect accent, n d It is random noise from the dialect.

[0053] Then, the first reference wake-up speech features can be input into a preset language classifier for binary classification, thereby identifying the language type of the first reference speech signal.

[0054] If the language type of the first reference speech signal is a dialect, then the content features of the wake-up speech features of the first sample can be used as a template to remove the content features of the wake-up words from the first reference wake-up speech features and separate the accent features when the user speaks the dialect.

[0055] In one embodiment of the present invention, step 104 may include the following steps:

[0056] Step 10411: Determine the intermediate speech features of the sample.

[0057] Step 10412: Determine the reference intermediate speech features.

[0058] Considering the hierarchical nature of speech feature networks, where accent information is mainly concentrated in specific dimensions of certain intermediate layers rather than across all layers and dimensions, this embodiment can select accent-sensitive dimensions through pre-experiments or lightweight statistics. Replacing all layers and dimensions with these dimensions not only eliminates redundant dimensions and slightly improves the purity of accent features (by about 5%), but also effectively reduces subsequent computational load and memory usage (by about 60%), making it suitable for edge computing.

[0059] On the one hand, the intermediate speech features of the sample can be determined; the intermediate speech features of the sample are the output of the intermediate layer within a specified range in the speech feature network when the wake-up speech features of the first sample are extracted.

[0060] On the other hand, reference intermediate speech features can be determined; the reference intermediate speech features are the outputs of the intermediate layer within a specified range in the speech feature network when extracting the first reference wake-up speech features.

[0061] Specifically, a specified range of intermediate layers can be set for different types of speech feature networks. For example, the specified range of intermediate layers can be set to layers 7-9 for Wav2Vec2Base and layers 6-9 for Hubert Base. In addition, for lightweight edge models with a total of 6-8 layers, the specified range of intermediate layers can usually be set to layers 4-5.

[0062] Step 10413: For the same intermediate layer, calculate the difference between the intermediate speech features of the sample and the intermediate speech features of the reference in each dimension, and use it as the intermediate residual.

[0063] Step 10414: Calculate the ratio between the variance of the intermediate residuals and the accent stability to obtain the discrimination index.

[0064] Iterate through each intermediate layer, and for the same intermediate layer, calculate the difference between the intermediate speech features of the sample and the intermediate speech features of the reference in each dimension, as the intermediate residual.

[0065] The discriminant is obtained by calculating the ratio between the variance of the intermediate residuals and the accent stability; where accent stability is the average similarity between the intermediate speech features of each pair of samples.

[0066] Then, the discrimination S of the l-th intermediate layer l It can be represented as S l =Var(r l1 ,r l2 ,……,r li ,……,r lm ) / Sim(d kwl1 ,d kwl2 ,……,d kwli ,……,d kwlm ), r li =d kwli -p li , where r li Let d be the intermediate residual of the i-th dimension of the l-th intermediate layer. kwli p represents the intermediate speech features of the i-th sample in the l-th intermediate layer. li Let Var be the i-th reference intermediate speech feature of the l-th intermediate layer, where Var is the variance. The larger the value, the greater the accent variation of the intermediate layer. Sim is the average similarity between the intermediate speech features of each pair of samples. The higher the value, the more stable the accent commonality of the intermediate layer.

[0067] Step 10415: Replace the first sample wake-up speech feature with the intermediate speech features of the multi-dimensional samples with the highest discriminative power, and replace the first reference wake-up speech feature with the intermediate speech features of the multi-dimensional references with the highest discriminative power.

[0068] On the one hand, the intermediate speech features of multiple dimensions with the highest discriminative power in each intermediate layer are concatenated, and the concatenated intermediate speech features of multiple dimensions replace the wake-up speech features of the first sample.

[0069] On the other hand, the multiple reference intermediate speech features with the highest discriminative power in each intermediate layer are concatenated, and the concatenated multiple reference intermediate speech features replace the first reference wake-up speech features.

[0070] After initially identifying the intermediate layers and dimensions that are sensitive to user accents, the intermediate layers and dimensions can be solidified. Subsequently, the intermediate speech features of the samples under the corresponding intermediate layers and dimensions can be replaced with the first sample wake-up speech features by means of table lookup, and the reference intermediate speech features under the corresponding intermediate layers and dimensions can be replaced with the first reference wake-up speech features.

[0071] In one embodiment of the present invention, step 104 may include the following steps:

[0072] Step 10421: Calculate the overall residual between the wake-up speech features of the first sample and the wake-up speech features of the first reference.

[0073] Initially, the first sample wake-up speech features and the first reference wake-up speech features in each dimension can be traversed. The differences between the first reference wake-up speech features and the first sample wake-up speech features in each dimension can be calculated to obtain the overall residual.

[0074] At this point, the set R of the total residuals can be represented as R = {r l ,r2,……,r i ,……,r m}, r i =d kwi -p i , where r i Let d be the total residual in dimension i. kwi p is the first reference wake-up speech feature in the i-th dimension. i The first sample of the i-th dimension is the wake-up speech feature.

[0075] Step 10422: Calculate the shrinkage covariance matrix of the total residuals.

[0076] In practical applications, there are few samples of users speaking dialects, and the principal components of PCA (Principal Component Analysis) are easily affected by noise, making the estimation of the covariance matrix unstable.

[0077] In this embodiment, the empirical covariance matrix of the overall residuals can be calculated, and the average value of the empirical covariance matrix can be merged with the empirical covariance matrix using a preset shrinkage coefficient to form a shrinkage covariance matrix.

[0078] At this point, the covariance matrix COV shrinks. shrink It can be represented as: COV shrink =λμI+(1-λ)COV emp Where λ is the contraction coefficient (e.g., 0.1-0.3), COV emp Let μI be the empirical covariance matrix, and μI be the mean of the empirical covariance matrix.

[0079] Step 10423: Perform random singular value decomposition on the contracted covariance matrix to obtain the accent vector space.

[0080] In this embodiment, the shrinkage covariance matrix can be subjected to random singular value decomposition to achieve rapid dimensionality reduction and obtain the accent vector space.

[0081] In a practical implementation, a random test matrix M can be generated, and Y = COV can be calculated. shrink M, perform QR decomposition on Y to obtain the Q matrix, and calculate B=Q. T ShrinkQ and perform standard SVD (singular value decomposition) on B to obtain the left singular vector matrix U. B Singular values ​​S and right singular vector matrix V B Calculate the accent vector space W=QU B .

[0082] Step 10424: Project the first reference wake-up speech features of each dimension into the orthogonal complement space of the accent vector space to obtain the de-accented features.

[0083] In this embodiment, the first reference wake-up speech features of each dimension can be projected onto the orthogonal complement space of the accent vector space to achieve content projection and obtain the content estimate after removing the dialect accent, which is denoted as the de-accenting feature.

[0084] At this point, the i-th dimension de-accenting feature c ti It can be represented as c ti =(IW (t-1) W T (t-1) )d kwi , where d kwi W represents the first reference wake-up speech feature in the i-th dimension. (t-1) Let I be the accent vector space at time t-1, where I is the identity matrix (constructing an orthogonal projection) and T is the transpose matrix.

[0085] Step 10425: Subtract the average value of the de-accented features from the first reference wake-up speech features to update the overall residual.

[0086] In this embodiment, for each dimension, the average value of the de-accented features is subtracted from the first reference wake-up speech features to update the overall residual.

[0087] At this point, the i-th dimension of the overall residual r ti It can be represented as: r ti =d kwi -mean(c t1 ,c t2 ,……, c ti ,……,c tm ), where c ti For the i-th dimension de-accenting feature, d kwi Let be the first reference wake-up speech feature in the i-th dimension, and mean be a function for calculating the average value.

[0088] Step 10426: Calculate the change in accent between the accent vector spaces before and after updating the overall residual.

[0089] In this embodiment, the updated overall residual can be used to update the accent vector space, and the accent vector space W corresponding to the overall residual before the update can be calculated. (t-1) The accent vector space W corresponding to the overall residual (t-1) The differences between them (such as the Frobenius norm) are used to obtain the range of accent variation.

[0090] Step 10427: If the accent change amplitude is less than the preset vector threshold, then determine the accent vector space after updating the overall residual as the accent features when the user speaks the dialect.

[0091] The accent change amplitude is compared with a preset vector threshold. If the accent change amplitude is less than the vector threshold, it means that the accent vector space has converged. This prevents abnormal samples (such as noise samples or false wake-up samples) from causing abrupt changes in accent features. The accent vector space after updating the overall residual is determined to be the accent features when the user speaks a dialect, thus maintaining the stability of the wake-up operation.

[0092] In practical applications, the residual between dialect wake words and non-dialect wake words may contain a small amount of wake word content deviation, such as content feature shifts caused by factors such as the user's personalized speech rate and personalized tone. In this embodiment, by iterating content projection and updating the vector space, components related to non-dialect content are gradually removed from the residual, while retaining the commonalities of pure accents related to direction.

[0093] In one embodiment of the present invention, step 104 may include the following steps:

[0094] Step 10431: Calculate the individual residuals between the newly added first reference wake-up speech features and the first sample wake-up speech features.

[0095] In non-initial situations, the shrinking write variance matrix can be continuously and incrementally updated based on the newly added first reference speech signal, reducing the frequency of full updates. This reduces resource consumption while continuously optimizing dialect-related accent features, making it suitable for operation on the edge.

[0096] In this embodiment, the difference between the newly added first reference wake-up speech feature and the first sample wake-up speech feature can be calculated to obtain the individual residual.

[0097] At this point, the individual residual r new It can be represented as: r new =d kw,new -p, where d kw,new p represents the newly added first reference wake-up speech feature, and p represents the first sample wake-up speech feature.

[0098] Step 10432: Update the overall residuals based on the individual residual values.

[0099] In this embodiment, the overall residual can be updated based on the individual residual value corresponding to the newly added first reference wake-up voice feature.

[0100] In the specific implementation, the first residual variation range is determined; wherein, the first residual variation range is the difference between the average value of the individual residual and the average value of the total residual, and the average value of the total residual represents the offset of the existing total residual on the whole.

[0101] Determine the new coefficient; the new coefficient is the reciprocal of the sum of the total number of residuals in the set plus 1.

[0102] The total residuals are updated by adding the product of the new coefficient and the magnitude of the residual change to the average value of the total residuals. The updated total residuals can then be added to the set.

[0103] At this point, the updated total residual r new,avg It can be represented as: r old,avg +1 / (m+1)×(r new -r old,avg ), where r old,avg r is the average of the total residuals. new Let m be the individual residual, and m be the total number of residuals in the set.

[0104] Step 10433: Update the shrinking covariance matrix based on the updated overall residuals.

[0105] In this embodiment, the shrinkage covariance matrix can be further updated based on the updated overall residual. Then, steps 10423-10427 are executed using the updated shrinkage covariance matrix to continuously optimize the accent features of the user when speaking dialect, thereby improving the subsequent wake-up success rate.

[0106] In the specific implementation, the attenuation coefficient is determined; the attenuation coefficient is the ratio between the difference between the number of total residuals in the set and 1, and the number of total residuals in the set.

[0107] Determine the magnitude of the second residual change; the magnitude of the second residual change is the difference between the individual residual and the updated population residual. The magnitudes of the first and second residual changes can quantify the impact of the new samples on the overall dimensional correlation.

[0108] The product of the attenuation coefficient and the shrinkage covariance matrix before the update is added to the product of the new coefficient, the change magnitude of the first residual, and the transpose of the change magnitude of the second residual, in order to update the shrinkage covariance matrix.

[0109] At this point, the updated shrinkage covariance matrix COV new It can be represented as: COV new =(m-1) / m×COV old +1 / (m+1)×(r new -r old,avg )(r new -r new,avg ) T , of which COV old The shrinkage covariance matrix before the update, r new For individual residuals, r old,avg r is the average of the total residuals. new,avg Let m be the updated total residual, m be the number of total residuals in the set, and T be the transpose matrix.

[0110] In this embodiment, the separation of accent features mainly uses covariance matrix, singular value decomposition, etc., which consumes less resources and is suitable for edge computing.

[0111] Step 105: Based on the speech feature network, the accent features are integrated into the wake-up speech features of the first sample to obtain the wake-up speech features of the second sample.

[0112] In this embodiment, accent features can be input at appropriate locations based on the structure of the speech feature network. The accent features are then incorporated into the process of extracting the first sample wake-up speech features to obtain the second sample wake-up speech features. That is, the second sample wake-up speech features contain both the information of the first sample wake-up speech features and the information of the accent features, thereby achieving directional fine-tuning of the first sample wake-up speech features (the user speaks a dialect).

[0113] In one fusion method, such as Figure 3 As shown, the structure of the speech feature network includes an encoder, a decoder, and a linear layer. The encoder is responsible for encoding the speech signal, the decoder is responsible for decoding the features output by the encoder, and the linear layer is responsible for linearly mapping the features output by the decoder.

[0114] The encoder and decoder are abstract structures, and the specific levels of the encoder and decoder may differ in different speech feature networks.

[0115] In this fusion method, the sample-coded speech features can be determined; wherein, the sample-coded speech features are the output of the encoder when the first sample wake-up speech features are extracted.

[0116] The Concat function is used to concatenate the accent feature F with the sample encoded speech feature to form the original multi-speech feature.

[0117] The original path between the encoder and decoder is temporarily truncated. The original multi-speech features are input into a pre-set bidirectional long short-term memory network (Bi-LSTM) to extract candidate multi-speech features. The feature evolution process from dialect to non-dialect and from non-dialect to dialect is learned to improve the quality of features and maintain the consistency of feature dimensions.

[0118] The candidate multi-speech features are input into the decoder and decoded into the target multi-speech features.

[0119] The target multi-speech features are input into a linear layer and mapped to second-sample wake-up speech features.

[0120] In this embodiment, a bidirectional long short-term memory network is added to the original speech feature network to fuse accent features. This consumes fewer resources and is suitable for on-device computation.

[0121] Step 106: Upon receiving the target speech signal, perform a wake-up operation based on the target speech signal, the first sample wake-up speech features, and the second sample wake-up speech features.

[0122] The electronic device continuously listens to voice signals while in standby mode. When it receives a target voice signal (i.e. the latest voice signal), it can perform a wake-up operation based on the target voice signal, the first sample wake-up voice features, and the second sample wake-up voice features.

[0123] In the specific implementation, the target speech signal is input into a speech feature network to extract the target wake-up speech features.

[0124] Calculate the similarity (e.g., cosine similarity) between the target wake-up speech features and the wake-up speech features of the first sample.

[0125] If the similarity between the target wake-up speech feature and the first sample wake-up speech feature is greater than or equal to the first confidence level and less than the second confidence level, then the similarity between the target wake-up speech feature and the second sample wake-up speech feature (such as cosine similarity) is calculated.

[0126] If the similarity between the target wake-up speech feature and the first sample wake-up speech feature is greater than or equal to the first confidence level and less than the second confidence level, and the similarity between the target wake-up speech feature and the second sample wake-up speech feature is greater than or equal to the second confidence level, then the wake-up operation is performed.

[0127] In this way, by using the wake-up voice features of the first sample as the initial identification threshold and the wake-up voice features of the second sample as the secondary identification threshold, the system can adapt to the changes in accent when a user speaks a dialect while ensuring the security of personalized wake-up for the user.

[0128] In this embodiment, upon receiving a sample voice signal entered by the user as a wake-up word, the sample voice signal is input into a voice feature network to extract the first sample wake-up voice feature. If the sample voice signal is not a dialect, the user is monitored for repeated wake-up events. A repeated wake-up event occurs when a wake-up operation succeeds after a previous failure. The first reference wake-up voice feature extracted from the first reference voice signal at the time of the failed wake-up operation is queried. If the first reference voice signal is a dialect, the user's accent features when speaking the dialect are separated from the first reference wake-up voice feature based on the first sample wake-up voice feature. The accent features are then integrated into the first sample wake-up voice feature based on the voice feature network to obtain the second sample wake-up voice feature. Upon receiving a target voice signal, a wake-up operation is performed based on the target voice signal, the first sample wake-up voice feature, and the second sample wake-up voice feature. This embodiment classifies the accent features of dialects from non-dialect wake-up voices, thereby fine-tuning the voice features of the wake-up word online. While maintaining a personalized wake-up word, the wake-up word adapts to the user's accent changes when speaking a dialect, thus improving the success rate of waking up the device using the wake-up word.

[0129] Furthermore, this embodiment is relatively simple to operate, suitable for on-device computation, and can ensure user privacy and security.

[0130] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0131] Reference Figure 4 The diagram illustrates a fine-tuning device for a wake-up word according to an embodiment of the present invention, which may specifically include the following modules:

[0132] The first sample feature extraction module 401 is used to input the sample voice signal into the voice feature network to extract the first sample wake-up voice features when it receives a sample voice signal entered by the user as a wake-up word.

[0133] The repeated wake-up event monitoring module 402 is used to monitor repeated wake-up events for the user if the sample voice signal is not a dialect; the repeated wake-up event is a successful wake-up operation after a failed wake-up operation.

[0134] The reference feature query module 403 is used to query the first reference wake-up voice feature extracted from the voice feature network when the wake-up operation fails.

[0135] The accent feature separation module 404 is used to separate the accent features of the user when speaking a dialect from the first reference wake-up speech features based on the first sample wake-up speech features if the first reference speech signal is a dialect.

[0136] The second sample feature generation module 405 is used to integrate the accent features into the first sample wake-up speech features according to the speech feature network to obtain the second sample wake-up speech features.

[0137] The wake-up operation execution module 406 is used to perform a wake-up operation based on the target voice signal, the first sample wake-up voice feature and the second sample wake-up voice feature when a target voice signal is received.

[0138] In one embodiment of the present invention, the repeated wake-up event listening module 402 is further configured to:

[0139] Upon receiving the first reference speech signal, the first reference speech signal is input into the speech feature network to extract the first reference wake-up speech feature;

[0140] If the similarity between the first reference wake-up voice feature and the first sample wake-up voice feature is greater than or equal to the first confidence level and less than the second confidence level, then the wake-up operation is determined to have failed, and a timer is started.

[0141] When a second reference voice signal is received before the start timer expires, the second reference voice signal is input into the voice feature network to extract the second reference wake-up voice feature.

[0142] If the similarity between the second reference wake-up voice feature and the first sample wake-up voice feature is greater than or equal to the second confidence level, then a wake-up operation is performed to determine that a duplicate wake-up event has occurred.

[0143] In one embodiment of the present invention, the accent feature separation module 404 is further configured to:

[0144] Calculate the overall residuals in each dimension between the wake-up speech features of the first sample and the wake-up speech features of the first reference;

[0145] Calculate the shrinkage covariance matrix of the total residuals;

[0146] Random singular value decomposition is performed on the contraction covariance matrix to obtain the accent vector space;

[0147] The first reference wake-up speech features are projected into the orthogonal complement space of the accent vector space to obtain the de-accented features;

[0148] The average value of the de-accented features is subtracted from the first reference wake-up speech features in each dimension to update the overall residual;

[0149] Calculate the magnitude of accent change between the accent vector space before and after updating the overall residual;

[0150] If the change in accent is less than a preset vector threshold, then the accent vector space after updating the overall residual is determined to be the accent feature of the user when speaking a dialect.

[0151] In one embodiment of the present invention, the accent feature separation module 404 is further configured to:

[0152] Determine the intermediate speech features of the sample; the intermediate speech features of the sample are the output of the intermediate layer within a specified range in the speech feature network when the wake-up speech features of the first sample are extracted;

[0153] Determine reference intermediate speech features; the reference intermediate speech features are the output of the intermediate layer within a specified range in the speech feature network when extracting the first reference wake-up speech features;

[0154] For the same intermediate layer, the differences between the sample intermediate speech features and the reference intermediate speech features in each dimension are calculated as intermediate residuals;

[0155] The discriminant is obtained by calculating the ratio between the variance of the intermediate residuals and the accent stability; the accent stability is the average similarity between the intermediate speech features of each pair of samples.

[0156] The first sample wake-up speech feature is replaced by the sample intermediate speech feature with the multiple dimensions of the highest discriminative power, and the first reference wake-up speech feature is replaced by the reference intermediate speech feature with the multiple dimensions of the highest discriminative power.

[0157] In one embodiment of the present invention, the accent feature separation module 404 is further configured to:

[0158] Calculate the individual residuals between the newly added first reference wake-up speech features and the first sample wake-up speech features;

[0159] The population residual is updated based on the individual residual values;

[0160] The shrinkage covariance matrix is ​​updated based on the updated overall residual.

[0161] In one embodiment of the present invention, the accent feature separation module 404 is further configured to:

[0162] Determine the first residual variation range; the first residual variation range is the difference between the individual residual and the average value of the population residual;

[0163] Determine the new coefficient; the new coefficient is the reciprocal of the sum of the total number of residuals plus 1;

[0164] The total residuals are updated by adding the product of the new coefficient and the magnitude of the residual change to the average value of the total residuals.

[0165] In one embodiment of the present invention, the accent feature separation module 404 is further configured to:

[0166] Determine the attenuation coefficient; the attenuation coefficient is the ratio between the difference between the total number of residuals and 1, and the total number of residuals.

[0167] Determine the magnitude of the second residual change; the magnitude of the second residual change is the difference between the individual residual and the updated population residual;

[0168] The product of the newly added coefficient, the first residual change magnitude, and the transpose of the second residual change magnitude is added to the product between the attenuation coefficient and the shrinkage covariance matrix to update the shrinkage covariance matrix.

[0169] In one embodiment of the present invention, the speech feature network includes an encoder, a decoder, and a linear layer; the second sample feature generation module 405 is further configured to:

[0170] Determine the sample encoded speech features; the sample encoded speech features are the output of the encoder when the first sample wake-up speech features are extracted.

[0171] The accent features are concatenated with the sample encoded speech features to form the original multi-speech features;

[0172] The original multi-speech features are input into a pre-set bidirectional long short-term memory network to extract candidate multi-speech features;

[0173] The candidate multi-speech features are input into the decoder and decoded into target multi-speech features;

[0174] The target multi-speech features are input into the linear layer and mapped to second sample wake-up speech features.

[0175] In one embodiment of the present invention, the wake-up operation execution module 406 is further configured to:

[0176] The target speech signal is input into the speech feature network to extract the target wake-up speech features;

[0177] If the similarity between the target wake-up voice feature and the first sample wake-up voice feature is greater than or equal to the first confidence level and less than the second confidence level, and the similarity between the target wake-up voice feature and the second sample wake-up voice feature is greater than or equal to the second confidence level, then a wake-up operation is performed.

[0178] The present invention provides a wake word fine-tuning device, which can realize the steps in the aforementioned wake word fine-tuning method embodiments.

[0179] It should be noted that the module division in the various wake-word fine-tuning devices provided in the above embodiments is illustrative and only represents one logical functional division. In actual implementation, other division methods are also possible. Furthermore, the functional modules in the various embodiments of this invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0180] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the embodiments of the present invention can be embodied in the form of a computer program product, which is stored in a computer storage medium and includes several instructions to cause an electronic device or processor to execute all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned computer storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] Furthermore, the wake word fine-tuning device and wake word fine-tuning method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0182] Reference Figure 5 The diagram illustrates an electronic device according to an embodiment of the present invention. Figure 5 As shown, the electronic device in this embodiment of the invention includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the above-described wake-word fine-tuning method embodiment. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described wake-word fine-tuning device embodiment.

[0183] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which can be used to describe the execution process of the computer program in the electronic device.

[0184] The electronic device may be a desktop computer, a cloud server, or other computing device. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 This is merely one example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0185] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0186] The memory can be an internal storage unit of the electronic device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or will be output.

[0187] This invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the wake word fine-tuning method as described in the foregoing embodiments.

[0188] This invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the wake word fine-tuning method as described in the foregoing embodiments.

[0189] This invention also discloses a computer program product that, when run on a computer, causes the computer to execute the wake word fine-tuning method described in the foregoing embodiments.

[0190] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for fine-tuning wake words, characterized in that, include: When a sample voice signal is received as a wake-up word by the user, the sample voice signal is input into the voice feature network to extract the first sample wake-up voice feature. If the sample speech signal is not a dialect, then the user is monitored for repeated wake-up events; the repeated wake-up event occurs when the wake-up operation succeeds after the wake-up operation fails. The first reference wake-up speech feature extracted from the speech feature network when the first reference speech signal fails to query the wake-up operation. If the first reference speech signal is a dialect, then the accent features of the user speaking the dialect are separated from the first reference wake-up speech features based on the first sample wake-up speech features. The accent features are integrated into the first sample wake-up speech features based on the speech feature network to obtain the second sample wake-up speech features; Upon receiving the target voice signal, a wake-up operation is performed based on the target voice signal, the first sample wake-up voice feature, and the second sample wake-up voice feature.

2. The method according to claim 1, characterized in that, The listening for repeated wake-up events by the user includes: Upon receiving the first reference speech signal, the first reference speech signal is input into the speech feature network to extract the first reference wake-up speech feature; If the similarity between the first reference wake-up voice feature and the first sample wake-up voice feature is greater than or equal to the first confidence level and less than the second confidence level, then the wake-up operation is determined to have failed, and a timer is started. When a second reference voice signal is received before the start timer expires, the second reference voice signal is input into the voice feature network to extract the second reference wake-up voice feature. If the similarity between the second reference wake-up voice feature and the first sample wake-up voice feature is greater than or equal to the second confidence level, then a wake-up operation is performed to determine that a duplicate wake-up event has occurred.

3. The method according to claim 1, characterized in that, The step of separating the accent features of the user when speaking a dialect from the first reference wake-up speech features based on the first sample wake-up speech features includes: Calculate the overall residuals in each dimension between the wake-up speech features of the first sample and the wake-up speech features of the first reference; Calculate the shrinkage covariance matrix of the total residuals; Random singular value decomposition is performed on the contraction covariance matrix to obtain the accent vector space; The first reference wake-up speech features are projected into the orthogonal complement space of the accent vector space to obtain the de-accented features; The average value of the de-accented features is subtracted from the first reference wake-up speech features in each dimension to update the overall residual; Calculate the magnitude of accent change between the accent vector space before and after updating the overall residual; If the change in accent is less than a preset vector threshold, then the accent vector space after updating the overall residual is determined to be the accent feature of the user when speaking a dialect.

4. The method according to claim 3, characterized in that, The step of separating the accent features of the user when speaking a dialect from the first reference wake-up speech features based on the first sample wake-up speech features further includes: Determine the intermediate speech features of the sample; the intermediate speech features of the sample are the output of the intermediate layer within a specified range in the speech feature network when the wake-up speech features of the first sample are extracted; Determine reference intermediate speech features; the reference intermediate speech features are the output of the intermediate layer within a specified range in the speech feature network when extracting the first reference wake-up speech features; For the same intermediate layer, the differences between the sample intermediate speech features and the reference intermediate speech features in each dimension are calculated as intermediate residuals; The discriminant is obtained by calculating the ratio between the variance of the intermediate residuals and the accent stability; the accent stability is the average similarity between the intermediate speech features of each pair of samples. The first sample wake-up speech feature is replaced by the sample intermediate speech feature with the multiple dimensions of the highest discriminative power, and the first reference wake-up speech feature is replaced by the reference intermediate speech feature with the multiple dimensions of the highest discriminative power.

5. The method according to claim 3, characterized in that, The step of separating the accent features of the user when speaking a dialect from the first reference wake-up speech features based on the first sample wake-up speech features further includes: Calculate the individual residuals between the newly added first reference wake-up speech features and the first sample wake-up speech features; The population residual is updated based on the individual residual values; The shrinkage covariance matrix is ​​updated based on the updated overall residual.

6. The method according to claim 5, characterized in that, Updating the overall residual based on the individual residual values ​​includes: Determine the first residual variation range; the first residual variation range is the difference between the individual residual and the average value of the population residual; Determine the new coefficient; the new coefficient is the reciprocal of the sum of the total number of residuals plus 1; The total residuals are updated by adding the product of the newly added coefficient and the change magnitude of the first residual to the average value of the total residuals; The step of updating the shrinkage covariance matrix based on the updated overall residuals includes: Determine the attenuation coefficient; the attenuation coefficient is the ratio between the difference between the total number of residuals and 1, and the total number of residuals. Determine the magnitude of the second residual change; the magnitude of the second residual change is the difference between the individual residual and the updated population residual; The product of the attenuation coefficient and the shrinkage covariance matrix is ​​added to the transpose of the second residual change magnitude, the newly added coefficient, and the first residual change magnitude to update the shrinkage covariance matrix.

7. The method according to any one of claims 1-6, characterized in that, The speech feature network includes an encoder, a decoder, and a linear layer; the step of integrating the accent features into the first sample wake-up speech features based on the speech feature network to obtain the second sample wake-up speech features includes: Determine the sample encoded speech features; the sample encoded speech features are the output of the encoder when the first sample wake-up speech features are extracted. The accent features are concatenated with the sample encoded speech features to form the original multi-speech features; The original multi-speech features are input into a pre-set bidirectional long short-term memory network to extract candidate multi-speech features; The candidate multi-speech features are input into the decoder and decoded into target multi-speech features; The target multi-speech features are input into the linear layer and mapped to second sample wake-up speech features.

8. The method according to claim 2, characterized in that, The step of performing a wake-up operation based on the target speech signal, the first sample wake-up speech features, and the second sample wake-up speech features includes: The target speech signal is input into the speech feature network to extract the target wake-up speech features; If the similarity between the target wake-up voice feature and the first sample wake-up voice feature is greater than or equal to the first confidence level and less than the second confidence level, and the similarity between the target wake-up voice feature and the second sample wake-up voice feature is greater than or equal to the second confidence level, then a wake-up operation is performed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the wake word fine-tuning method as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the wake word fine-tuning method as described in any one of claims 1-8.

Images