Voice wake-up method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of data processing, in particular to a voice wake-up method and device, computer equipment, a storage medium and a program product. The method comprises the steps that voice data input by a user are acquired, feature extraction is carried out on the voice data through a target model, voice features corresponding to the voice data are obtained, and the target model is a pre-trained ASR model; based on a mapping relation table and a pronunciation dictionary of the target model, text features corresponding to the registered text are determined, the mapping relation table is constructed based on a hidden layer feature layer of the target model, and the hidden layer feature layer comprises a corresponding relation between syllables and hidden layer features; and determining similarity data of the voice features and the text features, and determining whether to execute the wake-up task according to the similarity data. By adopting the method, the awakening accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a voice wake-up method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] In the voice interaction control method, users can match the registered wake-up word through voice input, and then wake up the device to perform the corresponding task; because this method is very convenient, it has been widely used in many fields.

[0003] In the related technology, the text features of the registered text are determined based on the hidden features extracted from the TTS (Text-to-Speech) model, and then the voice encoder is used to extract the voice information input by the user to obtain the voice features. The similarity between the text features and the voice features is calculated, and whether the device needs to be woken up is determined based on the result of the similarity calculation.

[0004] However, in the method of the related art, since the text features and speech features are obtained through different models, the dimensions of their feature vectors cannot be aligned. Therefore, when the similarity comparison between the text features and the speech features is performed, there will be a large error, which will affect the accuracy of awakening. Summary of the invention

[0005] Based on this, it is necessary to provide a voice wake-up method, device, computer equipment, storage medium and program product to address the above technical problems.

[0006] In a first aspect, the present application provides a voice wake-up method, comprising:

[0007] Acquire voice data input by a user, and perform feature extraction on the voice data through a target model to obtain voice features corresponding to the voice data, wherein the target model is a pre-trained ASR model;

[0008] Determining text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of the target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0009] Determine similarity data between the voice feature and the text feature, and determine whether to execute a wake-up task according to the similarity data.

[0010] In one embodiment, the method further comprises:

[0011] Determine a hidden feature sequence corresponding to each syllable from the hidden feature layer of the target model;

[0012] Determine the character feature corresponding to each syllable based on the hidden layer feature sequence;

[0013] A mapping relationship table is constructed based on the correspondence between syllables and character features.

[0014] In one embodiment, the text feature includes a character feature of each character in the registered text, and the determining the text feature corresponding to the registered text based on the mapping relationship table and the pronunciation dictionary of the target model includes:

[0015] Determining the syllable of each character in the registered text based on a pronunciation dictionary of the target model, wherein the pronunciation dictionary represents a correspondence between characters and syllables;

[0016] The character feature of each character in the registered text is determined by searching the mapping relationship table based on the syllable of each character in the registered text.

[0017] In one embodiment, the speech feature includes a plurality of speech frames distributed along a time series, the text feature includes a character feature corresponding to each character in the registered text, and the determining of the similarity data between the speech feature and the text feature includes:

[0018] Determine a similarity sequence between each speech frame and the text feature, wherein the similarity sequence includes the similarity corresponding to the speech frame and each character feature in the text feature;

[0019] Based on the similarity sequences between the respective speech frames and the text features, similarity data between the speech features and the text features are determined.

[0020] In one embodiment, determining the similarity sequence between each speech frame and the text feature includes:

[0021] Each of the speech frames is used as a query vector, and the text features are used as key vectors to be input into a cross-attention module for processing, so as to obtain a similarity sequence cross-attention module between each speech frame and the text features.

[0022] In one embodiment, each of the speech frames is used as a query vector, and the text features are used as key vectors to be input into a cross attention module for processing, so as to obtain a similarity sequence between each speech frame and the text features, including:

[0023] Each of the speech frames is used as a query vector, and the text features are used as key vectors to be input into a cross-attention module for processing, so as to obtain an intermediate alignment result of the cross-attention module as a similarity sequence between each of the speech frames and the text features.

[0024] In one embodiment, determining whether to execute the wake-up task according to the similarity data includes:

[0025] Performing dimension-upgrading processing on the similarity data to obtain dimension-upgrading data;

[0026] Inputting the dimension-increased data into a classification model to obtain a classification result, wherein the classification result represents a wake-up success or a wake-up failure;

[0027] Whether to execute a wake-up task is determined based on the classification result.

[0028] In one embodiment, the training step of the classification model includes:

[0029] Constructing a training set, wherein the training set includes a plurality of training samples, and the training samples include training speech data and text labels;

[0030] For the training sample, the training speech data is processed by a target model to obtain training speech features, and the training text features corresponding to the text labels are determined based on the mapping relationship table and the pronunciation dictionary of the target model, wherein the target model is a pre-trained ASR model, and the mapping relationship table is determined based on the pronunciation dictionary and hidden layer features of the target model;

[0031] Calculating similarity between the training speech feature and the training text feature of each training sample to obtain similarity data of each training sample;

[0032] The initial model is trained based on the similarity data of each training sample to obtain a classification model.

[0033] In one embodiment, the training speech features include multiple speech frames distributed along a time series; the training samples include positive samples and negative samples, the training speech data of the positive samples correspond to the content of the text label, and the training speech data of the negative samples do not correspond to the content of the text label; the initial model is trained based on the similarity calculation data of each of the training samples to obtain a classification model, including:

[0034] Determine a first loss, a second loss, and a third loss, and construct a loss function based on the first loss, the second loss, and the third loss, wherein the first loss includes the maximum value of the probability that each speech frame matches the wake-up failure label in the training speech features of the negative sample; the second loss includes the maximum value of the probability that each speech frame matches the wake-up success label in the training speech features of the positive sample; the third loss includes the ratio of the position of the speech frame in the positive sample with the maximum probability of matching the wake-up success label in the time series relative to the entire time series of the positive sample;

[0035] The initial model is trained based on the similarity data of each training sample and the loss function to obtain a classification model.

[0036] In a second aspect, the present application also provides a voice wake-up device, including a voice processing module, a text processing module and a judgment module, wherein:

[0037] A speech processing module is used to obtain speech data input by a user and perform feature extraction on the speech data through a target model to obtain speech features corresponding to the speech data, wherein the target model is a pre-trained ASR model;

[0038] A text processing module, used to determine text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of the target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0039] The judgment module is used to determine the similarity data between the voice feature and the text feature, and determine whether to execute the wake-up task according to the similarity data.

[0040] In one embodiment, the voice wake-up device further includes a mapping relationship table construction module, wherein the mapping relationship table construction module is used to:

[0041] Determine a hidden feature sequence corresponding to each syllable from the hidden feature layer of the target model;

[0042] Determine the character feature corresponding to each syllable based on the hidden layer feature sequence;

[0043] A mapping relationship table is constructed based on the correspondence between syllables and character features.

[0044] In one embodiment, the mapping relationship table construction module is specifically used to:

[0045] Determining the syllable of each character in the registered text based on a pronunciation dictionary of the target model, wherein the pronunciation dictionary represents a correspondence between characters and syllables;

[0046] The character feature of each character in the registered text is determined by searching the mapping relationship table based on the syllable of each character in the registered text.

[0047] In one embodiment, the speech feature includes a plurality of speech frames distributed along a time series, the text feature includes a character feature corresponding to each character in the registered text, and the judgment module is specifically used to:

[0048] Determine a similarity sequence between each speech frame and the text feature, wherein the similarity sequence includes the similarity corresponding to the speech frame and each character feature in the text feature;

[0049] Based on the similarity sequences between the respective speech frames and the text features, similarity data between the speech features and the text features are determined.

[0050] In one embodiment, the determination module is specifically used to:

[0051] Each of the speech frames is used as a query vector, and the text features are used as key vectors to be input into a cross-attention module for processing, so as to obtain a similarity sequence cross-attention module between each speech frame and the text features.

[0052] In one embodiment, the determination module is specifically used for:

[0053] Each of the speech frames is used as a query vector, and the text features are used as key vectors to be input into a cross-attention module for processing, so as to obtain an intermediate alignment result of the cross-attention module as a similarity sequence between each of the speech frames and the text features.

[0054] In one embodiment, the determination module is specifically used to:

[0055] Performing dimension-upgrading processing on the similarity data to obtain dimension-upgrading data;

[0056] Inputting the dimension-increased data into a classification model to obtain a classification result, wherein the classification result represents a wake-up success or a wake-up failure;

[0057] Whether to execute a wake-up task is determined based on the classification result.

[0058] In one embodiment, the voice wake-up device further includes a training module, and the training module is specifically used for:

[0059] Constructing a training set, wherein the training set includes a plurality of training samples, and the training samples include training speech data and text labels;

[0060] For the training sample, the training speech data is processed by a target model to obtain training speech features, and the training text features corresponding to the text labels are determined based on the mapping relationship table and the pronunciation dictionary of the target model, wherein the target model is a pre-trained ASR model, and the mapping relationship table is determined based on the pronunciation dictionary and hidden layer features of the target model;

[0061] Calculating similarity between the training speech feature and the training text feature of each training sample to obtain similarity data of each training sample;

[0062] The initial model is trained based on the similarity data of each training sample to obtain a classification model.

[0063] In one embodiment, the training speech features include a plurality of speech frames distributed along a time series; the training samples are divided into positive samples and negative samples, the training speech data of the positive samples correspond to the content of the text labels, and the training speech data of the negative samples do not correspond to the content of the text labels; the training module is specifically used for:

[0064] Determine a first loss, a second loss, and a third loss, and construct a loss function based on the first loss, the second loss, and the third loss, wherein the first loss includes the maximum value of the probability that each speech frame matches the wake-up failure label in the training speech features of the negative sample; the second loss includes the maximum value of the probability that each speech frame matches the wake-up success label in the training speech features of the positive sample; the third loss includes the ratio of the position of the speech frame in the positive sample with the maximum probability of matching the wake-up success label in the time series relative to the entire time series of the positive sample;

[0065] The initial model is trained based on the similarity data of each training sample and the loss function to obtain a classification model.

[0066] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0067] Acquire voice data input by a user, and perform feature extraction on the voice data through a target model to obtain voice features corresponding to the voice data, wherein the target model is a pre-trained ASR model;

[0068] Determining text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of the target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0069] Determine similarity data between the voice feature and the text feature, and determine whether to execute a wake-up task according to the similarity data.

[0070] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0071] Acquire voice data input by a user, and perform feature extraction on the voice data through a target model to obtain voice features corresponding to the voice data, wherein the target model is a pre-trained ASR model;

[0072] Determining text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of the target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0073] Determine similarity data between the voice feature and the text feature, and determine whether to execute a wake-up task according to the similarity data.

[0074] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0075] Acquire voice data input by a user, and perform feature extraction on the voice data through a target model to obtain voice features corresponding to the voice data, wherein the target model is a pre-trained ASR model;

[0076] Determining text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of the target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0077] Determine similarity data between the voice feature and the text feature, and determine whether to execute a wake-up task according to the similarity data.

[0078] In the above-mentioned voice wake-up method, device, computer equipment, storage medium and program product, the voice features are obtained by extracting the voice data input by the user through the target model, and the text features of the registered text are obtained through the mapping relationship table and the pronunciation dictionary of the target model, and the mapping relationship table is constructed based on the hidden feature layer of the target model, that is, the voice features and text features are both obtained based on the same target model, so the vectors in the voice features and text features are aligned in dimension and have consistency, that is, the error is small; therefore, when the dimensionally aligned voice features and text features are calculated for similarity, more accurate similarity data can be obtained, and then based on the similarity data, it can be more accurately determined whether the wake-up task needs to be performed; based on the voice wake-up method in the present application, the accuracy of wake-up by voice is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0080] FIG1 is a diagram of an application environment of a voice wake-up method in one embodiment;

[0081] FIG2 is a schematic diagram of a flow chart of a voice wake-up method in one embodiment;

[0082] FIG3 is a schematic diagram of a process of constructing a mapping relationship table in one embodiment;

[0083] FIG4 is a logic diagram of determining similarity data in the related art;

[0084] FIG5 is a schematic diagram of a process for determining similarity data in one embodiment;

[0085] FIG6 is a logic diagram of determining similarity data in one embodiment;

[0086] FIG7 is a schematic diagram of the structure of similarity data (matrix) in one embodiment;

[0087] FIG8 is a schematic diagram of determining whether to execute a wake-up task in one embodiment;

[0088] FIG9 is a schematic diagram of a data processing flow of a detection model in another embodiment;

[0089] FIG10 is a schematic diagram of a flow chart of a classification model training method in another embodiment;

[0090] FIG11 is a schematic diagram of a process for determining similarity data in another embodiment;

[0091] FIG12 is a schematic diagram of a process for constructing a loss function in one embodiment;

[0092] FIG13 is a schematic diagram of the processing logic of the initial Mamba model on the similarity matrix in one embodiment;

[0093] FIG14 is a structural block diagram of a voice wake-up device in one embodiment;

[0094] FIG. 15 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0095] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0096] In order to more clearly explain the technical effect of the technical solution of the present application, before specifically introducing the technical solution of the embodiment of the present application, the technical background or technical evolution context on which the embodiment of the present application is based is first introduced. With the technological progress in the field of deep learning, the voice interaction control method has been widely used in many fields. The user inputs the voice corresponding to the set registration text, and then wakes up the computer device to perform the corresponding task.

[0097] In the voice wake-up process of the related technology, it is necessary to match the voice input by the user with the registered text (wake-up word) customized by the user. The matching steps are: extracting hidden features from the TTS (Text-to-Speech) model to determine the text features of the registered text, and then using a voice encoder to extract the voice fragments input by the user to obtain voice features, and then perform a similarity comparison between the text features and the voice features, and determine whether the device needs to be woken up based on the results of the similarity comparison. However, in this way, the text features and voice features are obtained through different models, and the dimensions of their feature vectors are not uniform. Therefore, this method will have a large error when performing a similarity comparison between the text features and the voice features, which will affect the accuracy of the wake-up.

[0098] The voice wake-up method provided in the embodiment of the present application can be applied to the application environment shown in FIG1. ​​The user needs to pre-set the registration text (wake-up word) in the computer device 102. When in use, the user's voice is collected by the computer device 102 to obtain voice data. The computer device 102 compares the voice data and the registration text through the program product corresponding to the voice wake-up method in the present application, and then determines whether to execute the wake-up task.

[0099] The computer device 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, vehicle-mounted hosts, Internet of Things devices, and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0100] In an exemplary embodiment, as shown in FIG. 2 , a voice wake-up method is provided, which is described by taking the method applied to the computer device 102 in FIG. 1 as an example, and includes the following steps 10 to 50, wherein:

[0101] Step 10: Acquire the voice data input by the user, and extract features of the voice data through the target model to obtain voice features corresponding to the voice data. The target model is a pre-trained ASR (Automatic Speech Recognition) model.

[0102] For the embodiment of the present application, voice data is data collected by a computer device through a microphone. When collecting data, the microphone converts the user's voice signal into a corresponding voice signal in the form of an electrical signal through sound-to-electric conversion, which is the voice data.

[0103] The target model is a pre-trained ASR model, that is, an Automatic Speech Recognition (Automatic Speech Recognition) model. Among them, the target model can be pre-trained using a Conformer structure, or it can be pre-trained using a Transformer structure or an RNN (Recurrent Neural Network) model structure. In the embodiment of the present application, the target model is an ASR model pre-trained based on a Conformer structure as an example. Since the Conformer structure combines the advantages of convolutional neural networks and Transformers, its ability to process local and global information is enhanced.

[0104] The ASR model can extract features from the input voice data (encoding process) to obtain voice features, and then match the voice features in the pronunciation dictionary through a search algorithm to obtain the text corresponding to the voice data as the output of the target model. In the embodiment of the present application, only the feature extraction module inside the target model is used to extract features from the voice data input by the user to obtain voice features, without the need to obtain text output.

[0105] Specifically, speech features include multiple speech frames distributed along a time series, each speech frame corresponds to a feature vector, and the feature vector represents the probability of the speech frame matching each character in the pronunciation dictionary, wherein the number of characters in the pronunciation dictionary represents the dimension of the feature vector; for example, an ASR model can recognize and detect 1,000 characters, then the dimension of the feature vector extracted by the feature extraction module of the ASR model is 1,000 dimensions.

[0106] Step 30: Determine text features corresponding to the registered text based on the mapping relationship table and the pronunciation dictionary of the target model. The mapping relationship table is constructed based on the hidden feature layer of the target model. The hidden feature layer includes the correspondence between syllables and hidden features.

[0107] For the embodiment of the present application, the target model is a pre-trained ASR model, and there is a pronunciation dictionary in the pre-trained ASR model, wherein the pronunciation dictionary represents the correspondence between syllables and characters, and this correspondence indicates the corresponding syllables contained in the characters when they are converted into speech, or in other words, indicates the syllables contained in the pronunciation of the characters. The hidden feature layer includes the features gradually learned by the ASR model during the pre-training process, and the hidden feature layer includes the correspondence between syllables and hidden features. The registered text is one or more characters pre-customized by the user. The syllables corresponding to the characters in the registered text can be determined through the pronunciation dictionary, and then the hidden features corresponding to the characters in the registered text can be determined through the mapping relationship table constructed based on the hidden feature layer; that is, the hidden features corresponding to each character in the registered text can be used as the text features corresponding to the registered text.

[0108] Step 50: Determine similarity data between the voice feature and the text feature, and determine whether to execute the wake-up task based on the similarity data.

[0109] For the embodiment of the present application, the similarity data between the voice feature and the text feature is determined by the set similarity calculation logic. When the similarity data meets the set conditions, it is considered that the content represented by the voice input by the user matches the content represented by the registered text, and then the computer device is determined to perform the wake-up task; if the similarity data does not meet the set conditions, it is considered that the content represented by the voice input by the user does not match the content represented by the registered text, and at this time, the computer device determines not to perform the wake-up task. Among them, the wake-up task can be the computer device performing corresponding feedback, such as causing the computer device to switch from one state to another; the wake-up task can also be causing the computer device to perform one or more operations, such as causing the computer device to start and run an APP (Application) program.

[0110] In an embodiment of the present application, the speech feature is obtained by extracting the speech data input by the user through the feature extraction module of the target model, and the text feature of the registered text is obtained through a mapping relationship table and the pronunciation dictionary of the target model, and the mapping relationship table is constructed based on the hidden feature layer of the target model, that is, the speech feature and the text feature are both obtained based on the same target model; therefore, the vectors in the speech feature and the text feature are aligned in dimension and have consistency, that is, the error is small; therefore, when the dimensionally aligned speech features and text features are calculated for similarity, more accurate similarity data can be obtained, and then based on the similarity data, it can be more accurately determined whether the wake-up task needs to be performed; that is, based on the speech wake-up method in the present application, the accuracy of wake-up by voice is improved.

[0111] Furthermore, in one embodiment, step 20 is further included before step 30; step 20 is a step of constructing a mapping relationship table. Referring to FIG. 3 , step 20 may specifically include steps 21 to 23, wherein:

[0112] Step 21: Determine the hidden feature sequence corresponding to each syllable from the hidden feature layer of the target model.

[0113] Specifically, for ASR models with different structures, the position of the hidden feature layer in the model structure may be different; in the embodiment of the present application, the method for determining the hidden feature layer from the ASR model is not specifically limited. For example, the first network layer before the output layer of the target model can be determined as the hidden feature layer. In the hidden feature layer, one syllable corresponds to one hidden feature sequence, wherein the hidden feature sequence includes at least one hidden feature.

[0114] Step 22: determining the character feature corresponding to each syllable based on the hidden layer feature sequence;

[0115] Step 23: construct a mapping relationship table based on the correspondence between syllables and character features.

[0116] In the hidden feature layer, a syllable may correspond to multiple hidden features; in order to enhance the representation of the hidden features of each syllable, in an embodiment of the present application, for a hidden feature sequence corresponding to each syllable, a character feature is determined based on all the hidden features in the hidden feature sequence as the character feature of the corresponding syllable, so that a correspondence between a syllable and a character feature can be established, and then a mapping relationship table is constructed based on the correspondence between the syllable and the character feature.

[0117] Specifically, in one possible implementation, all hidden features in a hidden feature sequence corresponding to a syllable may be arithmetically averaged to obtain an average hidden feature corresponding to each syllable; wherein the average hidden feature is the character feature of the corresponding syllable. In another possible implementation, one hidden feature may be selected from all hidden features in a hidden feature sequence corresponding to a syllable as the character feature of the syllable. Of course, other methods may also be used to determine a hidden feature from a hidden feature sequence corresponding to a syllable as the character feature of the syllable, which is not specifically limited in the embodiments of the present application.

[0118] Further, in step 30, based on the mapping relationship table and the pronunciation dictionary of the target model, the text features corresponding to the registered text are determined, including: based on the pronunciation dictionary of the target model, the syllable of each character in the registered text is determined, and the pronunciation dictionary represents the correspondence between the character and the syllable; then based on the syllable of each character in the registered text, the mapping relationship table is queried to determine the character features of each character in the registered text, and then the text features of the registered text are obtained, wherein the text features include the character features of each character in the registered text.

[0119] Referring to FIG4 , in the related art, the method of determining the similarity data of speech features and text features is to input the speech features and text features into the cross-attention module for processing; wherein the speech features are used as the key vector (Key), and each character feature in the text features is used as the query vector (Query). The cross-attention module may include a weight calculation module and a weighted summation module; wherein the calculation logic of the weight calculation module is to perform dot product processing on the query vector (Query) and the key vector (Key) to calculate the correlation data between the query vector and the key vector, and to obtain a correlation sequence after normalizing the obtained correlation data through the Softmax function, and the correlation sequence is the weight of the value vector (Values) corresponding to the key vector; the calculation logic of the weighted summation module is to multiply each value vector with the corresponding weight, and then sum the multiplication results of each value vector with the corresponding weight, and the obtained output matrix is ​​the similarity data of the speech features and the text features. .

[0120] In the related art, since each character feature in the text feature is used as a query vector and the voice feature is used as a key vector, that is, what is determined in the related art is the similarity between each character in the text feature and the voice feature. In real-time scenarios, voice data may be streaming data, so the related art needs to perform sliding window processing on the voice data, and the sliding window processing is in the form of overlapping windows. That is, in the related art, each character feature in the text feature is taken as an individual, and the voice feature of the voice data corresponding to a single window is taken as an individual, and then the similarity between the two individuals is determined; that is, in the related art, it may be necessary to repeatedly calculate the similarity between a part of the character features and the voice features, which will generate a large amount of calculation. In addition, since the registration text can be customized, the length of the registration text set by each user is different. In order to improve the accuracy of wake-up, a larger window is usually set, thereby further increasing the amount of calculation. In practice, due to the limited computing power of computer equipment, the increase in the amount of calculation often leads to an increase in time consumption, that is, the method of calculating the similarity of text features and voice features in the related art will affect the efficiency of wake-up.

[0121] Therefore, in order to reduce the amount of calculation and time consumption, in one embodiment, in step 50, determining the similarity data between the speech feature and the text feature may specifically include steps 51 and 52, referring to FIG. 5, wherein:

[0122] Step 51, determining a similarity sequence between each speech frame and the text feature, wherein the similarity sequence includes the similarity corresponding to each character feature in the speech frame and the text feature;

[0123] Step 52: Determine similarity data between speech features and text features based on similarity sequences between each speech frame and the text features.

[0124] Specifically, referring to FIG6, for each speech frame, the similarity between the speech frame and each character feature in the text feature is determined, and then the similarity between the speech frame and each character feature in the text feature is used to form a set of similarity sequences. Specifically, the cross attention module takes each speech frame as a query vector (Query), and inputs the text feature as a key vector (Key) into the cross attention module for processing, and obtains the intermediate alignment result of the cross attention module as the similarity sequence between each speech frame and the text feature. Among them, the intermediate alignment result refers to alignment, and alignment is a correlation sequence obtained by the weight calculation module of the cross attention module. That is, in the embodiment of the present application, a speech frame is taken as a query vector (Query), and the text feature is input into the cross attention module as a key vector (Key); the correlation matrix obtained by normalizing the correlation between the query vector and the key vector after the weight calculation module calculates the correlation between the query vector and the key vector is obtained, and the correlation matrix is ​​used as the similarity sequence between the speech frame and the text feature. The similarity sequence of each speech frame and the text feature is arranged according to the time series corresponding to the speech frame, and a similarity matrix can be constructed, and the similarity matrix is ​​the similarity data of the speech feature and the text feature.

[0125] In one example, the composition of the similarity sequence is shown in Figure 7, where the speech features include speech frames 1, 2, 3 and 4, and the text features include character features A, B and C, wherein the similarity sequence between speech frame 1 and text features includes similarity 1A between speech frame 1 and character feature A, similarity 1B between speech frame 1 and character feature B, and similarity 1C between speech frame 1 and character feature C; the similarity of each speech frame constitutes a similarity matrix according to the time series of the speech frame.

[0126] Referring to Figures 6 and 7, in the embodiment of the present application, each speech frame in the speech feature is used as a query vector (Query), and the text feature is used as a key vector (Key). Therefore, in the cross-attention module, what is determined is the dot product of the key vector and each query vector, that is, the similarity sequence between the text feature and each speech frame. That is, in the scheme of the embodiment of the present application, each speech frame in the speech data is taken as an individual, and the text feature is taken as an individual, and then the similarity sequence between the two individuals is determined; therefore, the scheme of the embodiment of the present application can process each speech frame separately, and when facing streaming speech data, each speech frame can be processed in real time without sliding window processing, thereby reducing the amount of calculation and time consumption; that is to say, the similarity calculation method for text features and speech features in the embodiment of the present application can improve the efficiency of wake-up.

[0127] Further, referring to FIG8 , in step 50, determining whether to execute the wake-up task according to the similarity data may specifically include steps 53 to 55, wherein:

[0128] Step 53: Perform dimension-upgrading processing on the similarity data to obtain dimension-upgraded data.

[0129] Specifically, for registered texts of different lengths, different lengths of voice features and text features will be generated, that is, the similarity data may be similarity matrices with different numbers of rows and columns. In order to achieve the unification of dimensions, a larger target dimension can be pre-set in the embodiment of the present application, and then the dimension-upgraded data can be obtained by dimension-upgrading the similarity matrix, and the similarity matrix after dimension-upgrading is the target dimension.

[0130] In the embodiment of the present application, the similarity matrix is ​​increased in dimensionality by duplication, thereby realizing the dimensionality increase processing of the similarity matrix and obtaining the dimensionality increase data. Of course, the similarity matrix can also be increased in dimensionality by other methods, which are not specifically limited in the embodiment of the present application.

[0131] Step 54: input the dimension-increased data into a classification model to obtain a classification result, which represents a wake-up success or a wake-up failure;

[0132] Step 55: Determine whether to execute the wake-up task based on the classification result.

[0133] Specifically, the dimension-upgraded data is input into the trained classification model to obtain the classification result; that is, whether the text features and the voice features match is determined based on the classification model; wherein, when the text features and the voice features match, the classification result output is true (True), indicating a successful awakening; when the text features and the voice features do not match, the classification result output is false (False), indicating a failed awakening. The classification model can be obtained by training based on any neural network model. In the embodiment of the present application, the trained Mamba model is used as the classification model, which can directly output the classification result to determine whether the awakening is successful. For the specific training process of the classification model, please refer to the definition of the classification model in the following model training method embodiment.

[0134] Referring to FIG. 9 , in an embodiment of the present application, the program corresponding to the above-mentioned voice wake-up method is configured as a detection model, wherein: the voice data input by the user is processed by the feature extraction (encoding) module of the ASR model to obtain voice features; the registered text pre-set by the user is matched by the pronunciation dictionary of the ASR model and the mapping relationship table to obtain text features. The text features are used as key vectors and each voice frame in the voice features is input into the cross attention module as a query vector for dot product operation. The result of the dot product operation (the result of the middle alignment) is the similarity data (similarity matrix) between the voice features and the text features. The similarity data is processed by the dimension-upgrading module to obtain dimension-upgrading data; the dimension-upgrading data is input into the trained Mamba model for classification, and the output result of the Mamba model is the success or failure of the wake-up. The computer device determines whether to execute the wake-up task based on the output result of the Mamba model.

[0135] In an exemplary embodiment, as shown in FIG10 , a classification model training method is provided, which can be executed by a computer device or a server; the training method can specifically include steps 01 to 04, wherein:

[0136] Step 01: Construct a training set, which includes multiple training samples. The training samples include training speech data and text labels.

[0137] For the embodiment of the present application, the training set includes multiple training samples, wherein the training samples include positive samples and negative samples. The training speech data of the positive sample corresponds to the content of the text label, while the training speech data of the negative sample does not correspond to the content of the text label. For the ratio of positive samples to negative samples, no specific limitation is made in the embodiment of the present application.

[0138] Step 02: For the training samples, the training speech data is processed through the target model to obtain the training speech features, and the training text features corresponding to the text labels are determined based on the mapping relationship table and the pronunciation dictionary of the target model. The target model is a pre-trained ASR model, and the mapping relationship table is determined based on the pronunciation dictionary and hidden layer features of the target model.

[0139] Specifically, the training voice data is the data collected by the computer device through the microphone. When collecting, the microphone converts the user's voice signal into a corresponding voice signal in the form of an electrical signal through sound-to-electric conversion, which is the voice data. In an embodiment of the present application, the target model is a pre-trained ASR model; wherein the target model can be pre-trained using a Conformer structure, or can be pre-trained using a Transformer structure or an RNN (Recurrent Neural Network) structure. In an embodiment of the present application, the target model is an ASR model pre-trained based on a Conformer structure as an example. Since the Conformer structure combines the advantages of convolutional neural networks and Transformers, its ability to process local and global information is enhanced.

[0140] The ASR model can extract features from the input speech data (encoding process) to obtain speech features, and then match the speech features in the pronunciation dictionary through a search algorithm to obtain the text corresponding to the speech data as the output of the target model. In the embodiment of the present application, only the feature extraction module inside the target model is used to extract features from the training speech data to obtain the training speech features, without the need to obtain text output.

[0141] Specifically, the training speech features include multiple speech frames distributed along a time series, each speech frame corresponds to a feature vector, and a feature vector represents the probability of the speech frame matching each syllable in the pronunciation dictionary; wherein the number of characters in the pronunciation dictionary represents the dimension of the feature vector.

[0142] The target model is a pre-trained ASR model. The pre-trained ASR model contains a pronunciation dictionary, wherein the pronunciation dictionary represents the correspondence between syllables and characters. The hidden feature layer includes the features gradually learned by the ASR model during the pre-training process, and the hidden feature layer includes the correspondence between syllables and hidden features. The text label includes one or more characters. The syllable corresponding to the character in the text label can be determined through the pronunciation dictionary, and then the hidden features corresponding to the characters in the text label can be determined through the mapping relationship table constructed based on the hidden feature layer. The hidden features corresponding to the characters in the text label are the training character features of the text label, and the training text features include the character features corresponding to each character in the text label.

[0143] Step 03: Calculate the similarity between the training speech features and the training text features of each training sample to obtain the similarity data of each training sample;

[0144] Step 04: Train the initial model based on the similarity data of each training sample to obtain a classification model.

[0145] In the embodiment of the present application, for each training sample, the similarity calculation logic is used to calculate the similarity of the training speech features and the training text features, and the similarity data of each training sample is obtained; then the similarity data of each training sample and the classification label of the training sample are constructed as a set of training data to train the initial model. Among them, the classification label corresponding to the positive sample is true (True) to indicate a successful awakening, and the classification label corresponding to the negative sample is false (False) to indicate a failed awakening.

[0146] In an embodiment of the present application, the training speech features corresponding to each training sample are obtained by extracting the training speech data through the feature extraction module of the target model, and the training text features are obtained by matching the text labels through the mapping relationship table and the pronunciation dictionary of the target model, and the mapping relationship table is constructed based on the hidden feature layer of the target model, that is, the training speech features and the training text features are both obtained based on the same target model, so the vectors in the training speech features and the training text features are aligned in dimension and have consistency, that is, the error is small. Therefore, when the dimensionally aligned training speech features and training text features are calculated for similarity, more accurate similarity data can be obtained, and then the initial model can be trained based on the similarity data, so that the classification accuracy of the trained classification model can be higher.

[0147] Specifically, the initial model can be any deep learning model, such as an RNN (Recurrent Neural Network) model or a Mamba model; in the embodiment of the present application, the initial Mamba model is used as the initial model for training to obtain a classification model.

[0148] In one embodiment, in step 02, the training text features corresponding to the text tags are determined based on the mapping relationship table and the pronunciation dictionary of the target model, and before step 02, a step of constructing the mapping relationship table is also included. Further, the step of constructing the mapping relationship table may specifically include: determining the hidden feature sequence corresponding to each syllable from the hidden feature layer of the target model, determining the character feature corresponding to each syllable based on the hidden feature sequence, and then constructing the mapping relationship table based on the correspondence between the syllable and the character feature.

[0149] Specifically, for ASR models with different structures, the position of the hidden feature layer in the model structure may be different; in the embodiment of the present application, the method for determining the hidden feature layer from the ASR model is not specifically limited. For example, the first network layer before the output layer of the target model can be determined as the hidden feature layer. In the hidden feature layer, one syllable corresponds to one hidden feature sequence, wherein the hidden feature sequence includes at least one hidden feature.

[0150] In the hidden feature layer, one syllable may correspond to multiple hidden features; in order to enhance the representation of the hidden features of each syllable, in an embodiment of the present application, for the hidden feature sequence corresponding to each syllable, all the hidden features in the hidden sequence are arithmetically averaged to obtain the average hidden feature corresponding to each syllable; wherein the average hidden feature is the character feature of the corresponding syllable, so that a correspondence between a syllable and a character feature can be established, and then a mapping relationship table is constructed based on the correspondence between the syllable and the character feature.

[0151] Further, referring to FIG. 11 , step 03 may specifically include step 031 and step 032, wherein:

[0152] Step 031: for each training sample, determine a similarity sequence between each speech frame and the text feature, wherein the similarity sequence includes the similarity corresponding to each character feature in the speech frame and the text feature;

[0153] Specifically, for each training sample, each speech frame in the training speech feature is used as a query vector, and the training text feature is used as a key vector to be input into the cross-attention module for processing; specifically, the intermediate alignment result of the cross-attention module is used as the similarity sequence between each speech frame in the training speech feature and the training text feature. Among them, the intermediate alignment result refers to alignment, and alignment is a correlation sequence obtained by the weight calculation module of the cross-attention module. That is, in the embodiment of the present application, each speech frame is used as a query vector (Query), and the training text feature is used as a key vector (Key) to input the cross-attention module; the correlation matrix obtained by normalizing the query vector and the key vector after the weight calculation module calculates the correlation between the query vector and the key vector is obtained, and the correlation matrix is ​​used as the similarity sequence between the speech frame and the training text feature. The similarity sequence of each speech frame and the training text feature is arranged according to the time series corresponding to each speech frame in the training speech feature, and a similarity matrix can be constructed. The similarity matrix is ​​the similarity data between the training speech feature and the training text feature.

[0154] Step 032: For each training sample, based on the similarity sequences between each speech frame and the text feature, determine the similarity data between the speech feature and the text feature.

[0155] Specifically, since the speech frames in the training speech features are arranged in time series, each speech frame and the similarity sequence in the text features are arranged according to the time series of the speech frames to construct a similarity matrix, which is the similarity data corresponding to the training samples.

[0156] In the embodiment of the present application, each speech frame in the training speech feature is used as a query vector (Query), and the training text feature is used as a key vector (Key). Therefore, in the cross-attention module, what is determined is the dot product of the key vector and each query vector, that is, the similarity sequence between each speech frame and the training text feature. That is, in the scheme of the present application, each speech frame in the training speech data is taken as an individual, and the training text feature is taken as an individual, and then the similarity sequence between the two individuals is determined; therefore, the scheme of the embodiment of the present application can process each speech frame separately, and when facing streaming speech data, the speech frame can be processed in real time without sliding window processing, thereby reducing the amount of calculation and time consumption.

[0157] Referring to FIG. 12 , in step 04, before training the initial Mamba model, a step of constructing a loss function of the initial Mamba model is also included, which may specifically include steps 041-042, wherein:

[0158] Step 041, determine the first loss, the second loss and the third loss, and construct a loss function based on the first loss, the second loss and the third loss.

[0159] Specifically, referring to FIG. 13 , after the similarity sequence between any speech frame and text features is processed by the initial Mamba model, the initial Mamba model can obtain the probability Y1 that the speech frame matches the wake-up success label and the probability Y2 that the speech frame matches the wake-up failure label.

[0160] The first loss includes the maximum value of the probability that each speech frame matches the wake-up failure label in the training speech features of the negative sample; the second loss includes the maximum value of the probability that each speech frame matches the wake-up success label in the training speech features of the positive sample; the third loss includes the ratio of the position of the speech frame in the time series with the maximum probability of matching the wake-up success label in the positive sample relative to the entire time series. The loss function is constructed based on the first loss, the second loss, and the third loss.

[0161] Among them, the first loss is: ;

[0162] The second loss is: ;

[0163] The third loss is: .

[0164] Loss Function for: .

[0165] in, represents the speech features of negative samples, represents the speech features of the positive sample, Speech frames representing negative samples Wake-up failure label The probability of matching, Represents the speech frame of the positive sample Wake up successfully with tags Probability of matching; for The corresponding speech frame in the time series The position in is the time series of speech features, is the sequence number of the speech frame distributed along the time series in the speech feature of the negative sample, It is the sequence number of the speech frame in the speech feature of the positive sample along the time series distribution.

[0166] In fact, the loss function constructed based on the first loss and the second loss can also be used to train the initial Mamba model. However, in the embodiment of the present application, the third loss is taken into account to construct the loss function based on the first loss and the second loss, that is, the ratio of the position of the speech frame in the time series with the maximum probability of matching the wake-up success label in the positive sample to the entire time series. By adding the third loss, in the process of training the initial Mamba model, the model can continuously optimize and learn the wake-up point position in the voice data of each positive sample, so that the trained Mamba model (classification model) can output the wake-up result as soon as possible, thereby reducing the wake-up delay and improving the wake-up efficiency.

[0167] Step 042: Train the initial model based on the similarity data of each training sample and the loss function to obtain a classification model.

[0168] Furthermore, for text labels of different lengths, training speech features and training text features of different lengths will be generated, that is, the similarity data may be similarity matrices with different numbers of rows and columns. In order to achieve the unification of dimensions, a larger target dimension can be pre-set in the embodiment of the present application, and then the dimension-upgraded data can be obtained by dimension-upgrading the similarity matrix, and the similarity matrix after dimension-upgrading is the target dimension.

[0169] In the embodiment of the present application, the similarity matrix is ​​increased in dimensionality by duplication, thereby realizing the dimensionality increase processing of the similarity matrix and obtaining the dimensionality increase data. Of course, the similarity matrix can also be increased in dimensionality by other methods, which are not specifically limited in the embodiment of the present application.

[0170] By performing dimensionality-upgrading processing on the similarity data, the similarity data is unified to the same target dimension. Since the target dimension is set larger, it can adapt to registration texts of different lengths. The upgraded data is then used to train the initial Mamba model to obtain a classification model. In actual use, the classification model has stronger robustness in recognizing registration texts of different lengths.

[0171] Furthermore, the initial Mamba model is trained, and after the preset conditions are met, a trained classification model can be obtained; wherein the preset conditions may be that the training data corresponding to all training samples in the training set are trained to complete; or the training reaches a preset number of times; or the value of the loss function is within a preset range.

[0172] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0173] Based on the same inventive concept, the embodiment of the present application also provides a voice wake-up device for implementing the voice wake-up method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more voice wake-up device embodiments provided below can refer to the limitations on the voice wake-up method above, and will not be repeated here.

[0174] In an exemplary embodiment, as shown in FIG14 , a voice wake-up device is provided, including a voice processing module 1401, a text processing module 1402, and a judgment module 1403, wherein:

[0175] The speech processing module 1401 is used to obtain speech data input by the user and perform feature extraction on the speech data through a target model to obtain speech features corresponding to the speech data. The target model is a pre-trained ASR model.

[0176] A text processing module 1402 is used to determine text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of a target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0177] The judgment module 1403 is used to determine the similarity data between the voice feature and the text feature, and determine whether to execute the wake-up task according to the similarity data.

[0178] In one embodiment, the voice wake-up device further includes a mapping relationship table construction module, wherein the mapping relationship table construction module is used to:

[0179] Determine a hidden feature sequence corresponding to each syllable from the hidden feature layer of the target model;

[0180] Determine the character features corresponding to each syllable based on the hidden feature sequence;

[0181] A mapping relationship table is constructed based on the correspondence between syllables and character features.

[0182] In one embodiment, the mapping relationship table construction module is specifically used to:

[0183] Determine the syllable of each character in the registered text based on the pronunciation dictionary of the target model, and the pronunciation dictionary represents the correspondence between characters and syllables;

[0184] Based on the syllable of each character in the registered text, the character feature of each character in the registered text is determined by querying the mapping relationship table.

[0185] In one embodiment, the speech feature includes a plurality of speech frames distributed along a time series, the text feature includes a character feature corresponding to each character in the registered text, and the determination module 1403 is specifically configured to:

[0186] Determine a similarity sequence between each speech frame and the text feature, wherein the similarity sequence includes a similarity corresponding to each character feature in the speech frame and the text feature;

[0187] Based on the similarity sequences between each speech frame and the text feature, similarity data between the speech feature and the text feature are determined.

[0188] In one embodiment, the determination module 1403 is specifically configured to:

[0189] Each speech frame is used as a query vector, and the text features are used as key vectors to be input into the cross-attention module for processing, and the similarity sequence between each speech frame and the text features is obtained.

[0190] In one embodiment, the determination module 1403 is specifically used to:

[0191] Each speech frame is used as a query vector, and the text feature is used as a key vector to be input into the cross-attention module for processing, and the intermediate alignment result of the cross-attention module is obtained as the similarity sequence between each speech frame and the text feature.

[0192] Cross-Attention Module.

[0193] In one embodiment, the determination module 1403 is specifically configured to:

[0194] Performing dimension-upgrading processing on the similarity data to obtain dimension-upgrading data;

[0195] Inputting the dimension-increased data into a classification model to obtain a classification result, wherein the classification result represents a wake-up success or a wake-up failure;

[0196] Whether to execute the wake-up task is determined based on the classification result.

[0197] In one embodiment, the voice wake-up device further includes a training module, and the training module is specifically used for:

[0198] Construct a training set, which includes multiple training samples, and the training samples include training speech data and text labels;

[0199] For the training samples, the training speech data is processed by the target model to obtain the training speech features, and the training text features corresponding to the text labels are determined based on the mapping relationship table and the pronunciation dictionary of the target model. The target model is a pre-trained ASR model, and the mapping relationship table is determined based on the pronunciation dictionary and hidden layer features of the target model.

[0200] Calculate the similarity of the training speech features and the training text features of each training sample to obtain the similarity data of each training sample;

[0201] The initial model is trained based on the similarity data of each training sample to obtain a classification model.

[0202] In one embodiment, the training speech features include a plurality of speech frames distributed along a time series; the training samples include positive samples and negative samples, the training speech data of the positive samples correspond to the content of the text labels, and the training speech data of the negative samples do not correspond to the content of the text labels; the training module is specifically used for:

[0203] Determine a first loss, a second loss, and a third loss, and construct a loss function based on the first loss, the second loss, and the third loss, wherein the first loss includes the maximum value of the probability that each speech frame matches the wake-up failure label in the training speech features of the negative sample; the second loss includes the maximum value of the probability that each speech frame matches the wake-up success label in the training speech features of the positive sample; the third loss includes the ratio of the position of the speech frame in the positive sample with the maximum probability of matching the wake-up success label in the time series relative to the entire time series of the positive sample;

[0204] The initial model is trained based on the similarity data of each training sample and the loss function to obtain the classification model.

[0205] In the voice wake-up device in the embodiment of the present application, the voice features are obtained by extracting the voice data input by the user through the target model, and the text features of the registered text are obtained through the mapping relationship table and the pronunciation dictionary of the target model, and the mapping relationship table is constructed based on the hidden feature layer of the target model, that is, the voice features and the text features are both obtained based on the same target model, therefore, the vectors in the voice features and the text features are aligned in dimension and have consistency, that is, the error is small; therefore, when the dimensionally aligned voice features and text features are calculated for similarity, more accurate similarity data can be obtained, and then based on the similarity data, it can be more accurately determined whether the wake-up task needs to be performed; based on the voice wake-up method in the present application, the accuracy of wake-up by voice is improved.

[0206] Each module in the above-mentioned voice wake-up device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0207] In an exemplary embodiment, a computer device is provided, and the internal structure diagram of the computer device can be shown in Figure 15: the computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external computer device in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a voice wake-up method as in the above-mentioned voice wake-up method embodiment is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0208] Those skilled in the art will understand that the structure shown in FIG. 15 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0209] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0210] Acquire the voice data input by the user, and extract features from the voice data through the target model to obtain the voice features corresponding to the voice data. The target model is a pre-trained ASR model.

[0211] Determine text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of a target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0212] Determine similarity data between the voice feature and the text feature, and determine whether to perform a wake-up task based on the similarity data.

[0213] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0214] Acquire the voice data input by the user, and extract features from the voice data through the target model to obtain the voice features corresponding to the voice data. The target model is a pre-trained ASR model.

[0215] Determine text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of a target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0216] Determine similarity data between the voice feature and the text feature, and determine whether to perform a wake-up task based on the similarity data.

[0217] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0218] Acquire the voice data input by the user, and extract features from the voice data through the target model to obtain the voice features corresponding to the voice data. The target model is a pre-trained ASR model.

[0219] Determine text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of a target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features;

[0220] Determine similarity data between the voice feature and the text feature, and determine whether to perform a wake-up task based on the similarity data.

[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0223] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0224] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A voice wake-up method, characterized in that: The method comprises: Acquire voice data input by a user, and perform feature extraction on the voice data through a target model to obtain voice features corresponding to the voice data, wherein the target model is a pre-trained ASR model; Determining text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of the target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features; Determine similarity data between the voice feature and the text feature, and determine whether to execute a wake-up task according to the similarity data.

2. The method according to claim 1, characterized in that: The method further comprises: Determine a hidden feature sequence corresponding to each syllable from the hidden feature layer of the target model; Determine the character feature corresponding to each syllable based on the hidden layer feature sequence; A mapping relationship table is constructed based on the correspondence between syllables and character features.

3. The method according to claim 2, characterized in that The text feature includes a character feature of each character in the registered text, and the determining of the text feature corresponding to the registered text based on the mapping relationship table and the pronunciation dictionary of the target model includes: Determining the syllable of each character in the registered text based on a pronunciation dictionary of the target model, wherein the pronunciation dictionary represents a correspondence between characters and syllables; The character feature of each character in the registered text is determined by searching the mapping relationship table based on the syllable of each character in the registered text.

4. The method according to any one of claims 1 to 3, characterized in that: The speech feature includes a plurality of speech frames distributed along a time series, the text feature includes a character feature corresponding to each character in the registered text, and the determining of the similarity data between the speech feature and the text feature includes: Determine a similarity sequence between each speech frame and the text feature, wherein the similarity sequence includes the similarity corresponding to the speech frame and each character feature in the text feature; Based on the similarity sequences between the respective speech frames and the text features, similarity data between the speech features and the text features are determined.

5. The method according to claim 4, characterized in that The determining of the similarity sequence between each speech frame and the text feature comprises: Each of the speech frames is used as a query vector, and the text features are used as key vectors to be input into a cross-attention module for processing, so as to obtain a similarity sequence between each speech frame and the text features.

6. The method according to claim 5, characterized in that The method uses each of the speech frames as a query vector and inputs the text features as key vectors into a cross attention module for processing to obtain a similarity sequence between each speech frame and the text features, including: Each of the speech frames is used as a query vector, and the text features are used as key vectors to be input into a cross-attention module for processing, so as to obtain an intermediate alignment result of the cross-attention module as a similarity sequence between each of the speech frames and the text features.

7. The method according to claim 4, characterized in that The determining whether to execute the wake-up task according to the similarity data includes: Performing dimension-upgrading processing on the similarity data to obtain dimension-upgrading data; Inputting the dimension-increased data into a classification model to obtain a classification result, wherein the classification result represents a wake-up success or a wake-up failure; Whether to execute a wake-up task is determined based on the classification result.

8. The method according to claim 7, characterized in that The training steps of the classification model include: Constructing a training set, wherein the training set includes a plurality of training samples, and the training samples include training speech data and text labels; For the training sample, the training speech data is processed by the target model to obtain the training speech features, and the training text features corresponding to the text labels are determined based on the mapping relationship table and the pronunciation dictionary of the target model, wherein the target model is a pre-trained ASR model, and the mapping relationship table is determined based on the pronunciation dictionary and hidden layer features of the target model; Calculating similarity between the training speech feature and the training text feature of each training sample to obtain similarity data of each training sample; The initial model is trained based on the similarity data of each training sample to obtain a classification model.

9. The method according to claim 8, characterized in that The training speech features include a plurality of speech frames distributed along a time series; the training samples include positive samples and negative samples, the training speech data of the positive samples correspond to the content of the text labels, and the training speech data of the negative samples do not correspond to the content of the text labels; The initial Mamba model is trained based on the similarity calculation data of each training sample to obtain a classification model, including: Determine a first loss, a second loss, and a third loss, and construct a loss function based on the first loss, the second loss, and the third loss, wherein the first loss includes the maximum value of the probability that each speech frame matches the wake-up failure label in the training speech features of the negative sample; the second loss includes the maximum value of the probability that each speech frame matches the wake-up success label in the training speech features of the positive sample; the third loss includes the ratio of the position of the speech frame in the positive sample with the maximum probability of matching the wake-up success label in the time series relative to the entire time series of the positive sample; The initial model is trained based on the similarity data of each training sample and the loss function to obtain a classification model.

10. A voice wake-up device, characterized in that: The device includes a speech processing module, a text processing module and a judgment module, wherein: A speech processing module is used to obtain speech data input by a user and perform feature extraction on the speech data through a target model to obtain speech features corresponding to the speech data, wherein the target model is a pre-trained ASR model; A text processing module, used to determine text features corresponding to the registered text based on a mapping relationship table and a pronunciation dictionary of the target model, wherein the mapping relationship table is constructed based on a hidden feature layer of the target model, and the hidden feature layer includes a correspondence between syllables and hidden features; The judgment module is used to determine the similarity data between the voice feature and the text feature, and determine whether to execute the wake-up task according to the similarity data.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images