Voice wake-up recognition method and device

Abstract

The invention provides a voice wake-up recognition method and device. The method comprises the following steps: performing wake-up word recognition on wake-up voice data; under the condition that the wake-up word exists in the wake-up voice data, determining a target language to which the wake-up word in the wake-up voice data belongs, and cutting off the wake-up word from the wake-up voice data; and based on a target voice recognition model, performing voice recognition on the wake-up voice data after the wake-up word is cut off to obtain a voice wake-up recognition result, the target voice recognition model being used for recognizing voice data in a target language. According to the voice wake-up recognition method and device provided by the invention, the wake-up word recognition is firstly performed on the short, clear and easily recognized wake-up voice data, and the starting point of the wake-up intention is determined, so that the calculation amount can be reduced, a plurality of different wake-up words can be accurately recognized, and then the wake-up words in the wake-up voice data are cut off; therefore, the interference of the wake-up word on the subsequent recognition of the wake-up intention of the user can be avoided, and the voice wake-up recognition precision is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of voice wake-up recognition, and in particular to a voice wake-up recognition method and device. Background Art

[0002] Voice wake-up recognition is a technology that activates a device or system to enter an interactive state through specific voice commands. When the user says the preset wake-up word, the device wakes up from standby or low-power mode and is ready to receive subsequent commands from the user.

[0003] Currently, a larger recognition model is often used for voice wake-up recognition, which can not only recognize the wake-up words in voice commands, but also recognize the command information in voice commands. However, since the general recognition model is rarely trained to recognize specific wake-up words, it may mistake words similar to wake-up words mentioned by users in daily conversations as wake-up words, causing the device to be activated at inappropriate times. Summary of the invention

[0004] The present invention provides a voice wake-up recognition method and device to solve the defects in the prior art.

[0005] The present invention provides a voice wake-up recognition method, comprising the following steps: Perform wake-up word recognition on wake-up voice data; In the case where the wake-up word exists in the wake-up voice data, determining the target language to which the wake-up word in the wake-up voice data belongs, and removing the wake-up word from the wake-up voice data; Based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is cut off to obtain a speech wake-up recognition result, and the target speech recognition model is used to recognize the speech data in the target language.

[0006] According to a voice wake-up recognition method provided by the present invention, the step of determining the target language to which the wake-up word in the wake-up voice data belongs includes: Extracting voice features of the wake-up voice data; Encoding the speech feature to obtain a coded feature; Based on the encoding feature, determine the target language to which the wake-up word in the wake-up voice data belongs and the audio endpoint of the wake-up word.

[0007] According to a voice wake-up recognition method provided by the present invention, the wake-up word recognition is performed on the wake-up voice data, and when the wake-up word exists in the wake-up voice data, determining the target language to which the wake-up word in the wake-up voice data belongs includes: Based on the wake-up word recognition model, perform wake-up word recognition on the wake-up voice data, and if the wake-up word exists in the wake-up voice data, determine the target language to which the wake-up word in the wake-up voice data belongs and the audio endpoint of the wake-up word; The wake-up word recognition model is obtained by fine-tuning a pre-trained wake-up model based on first sample voice data in different languages, the language label of the sample wake-up word in each first sample voice data, and the audio endpoint label of the sample wake-up word. The pre-trained wake-up model is trained based on second sample voice data in different languages ​​and the voice text label of each second sample voice data.

[0008] According to a voice wake-up recognition method provided by the present invention, the voice recognition is performed on the wake-up voice data after the wake-up word is cut out based on the target voice recognition model, including: Based on the target language to which the wake-up word in the wake-up voice data belongs, determining a target voice recognition model from a plurality of voice recognition models, each voice recognition model being used to recognize voice data in a different language; Based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is cut out.

[0009] According to a voice wake-up recognition method provided by the present invention, the target voice recognition model includes an encoding module, a decoding module and an attention module of the target voice recognition model; the target voice recognition model is trained based on the following steps: Based on the encoding module, encoding the voice features of the third sample voice data to obtain a sample encoding feature, wherein the third sample voice data is the voice data after the wake-up word is cut off; Based on the attention module of the target speech recognition model, performing attention calculation on the sample coding feature to obtain a sample attention feature; Based on the decoding module, the sample attention feature is decoded to obtain a sample recognition text; Based on the difference between the sample recognition text and the speech text label corresponding to the third sample speech data, the initial model is trained to obtain the target speech recognition model.

[0010] According to a speech wake-up recognition method provided by the present invention, each speech recognition model shares the encoding module and the decoding module, and during the training process of any speech recognition model, the attention module parameters of other speech recognition models are frozen.

[0011] The present invention also provides a voice wake-up recognition device, comprising the following modules: A recognition unit, used to recognize the wake-up word of the wake-up voice data; a determining unit, configured to determine, when the wake-up word exists in the wake-up voice data, a target language to which the wake-up word in the wake-up voice data belongs, and to delete the wake-up word from the wake-up voice data; The wake-up unit is used to perform speech recognition on the wake-up speech data after the wake-up word is cut out based on a target speech recognition model to obtain a speech wake-up recognition result. The target speech recognition model is used to recognize the speech data in the target language.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the voice wake-up recognition method described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the voice wake-up recognition method as described above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the voice wake-up recognition method described above is implemented.

[0015] The voice wake-up recognition method and device provided by the present invention first recognize the wake-up word for the short, clear and easy-to-recognize wake-up voice data, and determine the starting point of the wake-up intention, which can not only reduce the amount of calculation, but also accurately identify multiple different wake-up words. In the case where there is a wake-up word in the wake-up voice data, the target language to which the wake-up word in the wake-up voice data belongs is determined, thereby ensuring that the target language recognition model matches the actual language requirements of the wake-up voice data, thereby improving the voice recognition accuracy. At the same time, the present invention cuts out the wake-up words in the wake-up voice data, thereby avoiding the interference of the wake-up words in the subsequent recognition of the user's wake-up intention, and further improving the voice wake-up recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the voice wake-up recognition method provided by the present invention.

[0018] Figure 2 It is a structural diagram of the speech wake-up word recognition model provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the speech recognition model provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the voice wake-up recognition device provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The optimization scheme based on the recognition model mainly uses a larger recognition model for voice wake-up recognition. This model can not only recognize the wake-up words in the voice commands, but also recognize the command information in the voice commands. However, since the general recognition model is rarely trained to recognize specific wake-up words, it may mistake words similar to the wake-up words mentioned by users in daily conversations as wake-up words, causing the device to be activated at inappropriate times. In addition, if a recognition model is maintained based on each wake-up word, the maintenance cost is too high, and maintaining a set of recognition models for multiple wake-up words will result in cross-talk recognition between wake-up words, such as the appearance of wake-up words of other cars on a certain car.

[0024] In addition, there are also recognition solutions based on pure wake-up models, which mainly rely on a relatively small wake-up model. The model is specially trained to recognize specific wake-up words or phrases and can run efficiently when the device is in a low-power state. Once the model detects the wake-up word, the device will switch to a higher power mode and prepare to receive and execute subsequent instructions from the user. Since pure wake-up models are usually relatively small, their ability to process complex instructions is limited, resulting in low recognition accuracy for subsequent instructions. In addition, due to the diversity and complexity of languages, pure wake-up models may have difficulty accurately identifying wake-up words in different languages, thereby reducing the user experience.

[0025] To this end, the present invention provides a voice wake-up recognition method. Figure 1 is a flow chart of the voice wake-up recognition method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0026] Step 110: Perform wake-up word recognition on the wake-up voice data.

[0027] Here, the wake-up voice data refers to the voice information containing a specific wake-up word issued by the user in order to activate the device or system. The wake-up voice data can be a voice command issued by the user when he wants to interact with the device, which is obtained by an audio acquisition device such as a microphone on the device and converted into wake-up voice data.

[0028] It should be noted that the wake-up voice data may include not only the wake-up word, but also the user's wake-up intention (such as the operation the user wants the device to perform). However, if the general recognition model in the traditional solution is used to recognize both the wake-up word and the wake-up intention, then since the general recognition model is less specially trained to recognize specific wake-up words, it may mistake words similar to the wake-up words mentioned by the user in daily conversation as wake-up words, causing the device to be activated at an inappropriate time. In addition, if a recognition model is maintained based on each wake-up word, the maintenance cost is too high, and if a set of recognition models is maintained for multiple wake-up words, cross-talk recognition between wake-up words will occur, such as the wake-up words of other cars appearing on a certain car.

[0029] Based on this, the embodiment of the present invention first performs wake-up word recognition on the wake-up voice data separately, that is, the focus is on whether there is a wake-up word in the wake-up voice data, without the need to immediately parse the user's complete wake-up intention or subsequent instructions. Since the wake-up word is usually a short, clear and easy-to-recognize word or phrase, the computational complexity of using the model to recognize it is small. In order to further reduce the amount of calculation, a lightweight model can be used to recognize the wake-up word. In addition, if there is a need to update the wake-up word, the model parameters can be optimized to adapt to the new wake-up word. There is no need to train a model for each wake-up word. Instead, one model can be used to maintain multiple wake-up words, thereby improving the flexibility and scalability of the model, so that the device can easily adapt to the wake-up word requirements of different users or different scenarios.

[0030] In addition, the wake-up word recognition can also be performed on the wake-up voice data based on the preset wake-up word matching method, such as by comparing the wake-up voice data with the preset wake-up word library to determine whether there is a matching wake-up word. If there is a need to update the wake-up word, the new wake-up word can also be adapted by updating the preset wake-up word. Updating the preset wake-up word usually only requires adding or replacing a new wake-up word in the device or system without retraining or adjusting the entire recognition model. This operation method not only saves time and resource costs, but also enables the device to easily adapt to the wake-up word needs of different users or different scenarios.

[0031] Step 120: When the wake-up word exists in the wake-up voice data, determine the target language to which the wake-up word in the wake-up voice data belongs, and delete the wake-up word from the wake-up voice data.

[0032] Specifically, when it is determined that there is a wake-up word in the wake-up voice data, it indicates that the user has a wake-up intention, and at this time, the user's wake-up intention can be focused on. Since the wake-up intention usually corresponds to a long text sentence, and users of different languages ​​may use different vocabulary, grammatical structure and intonation when expressing the same wake-up intention, these factors together constitute the complexity in speech recognition. If the same speech recognition model is used to recognize the wake-up intentions of different languages, due to the large differences in speech features, vocabulary, grammatical structure and intonation between different languages, the same speech recognition model may not be able to adapt to these differences at the same time, resulting in a decrease in recognition performance. In order to improve the accuracy of wake-up recognition, it is necessary to train and optimize speech recognition models for different languages ​​to ensure that the model can accurately understand and recognize the user's wake-up intention in different languages. In this regard, in the embodiment of the present invention, when it is determined that there is a wake-up word in the wake-up voice data, the target language to which the wake-up word in the wake-up voice data belongs is determined, and the target language is the language corresponding to the wake-up voice data, so that the corresponding speech recognition model can be set for the wake-up voice data in a targeted manner to improve the accuracy of speech recognition.

[0033] In addition, considering that the wake-up word is the starting signal for the user to interact with the device, once the recognition is successful, since the wake-up word itself does not contain the information required to perform a specific task, pruning the wake-up word can remove the redundant part in the wake-up voice data and avoid interference with the subsequent recognition of the user's wake-up intention. For example, in some cases, the wake-up word may be similar to the vocabulary of the user's intended instruction in the wake-up voice data, which may lead to misrecognition.

[0034] Exemplarily, when the wake-up word recognition is performed on the wake-up voice data in step 110, the position of the wake-up word may be marked, and the wake-up word may be deleted from the wake-up voice data based on the position.

[0035] Step 130: Based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is cut off to obtain a speech wake-up recognition result. The target speech recognition model is used to recognize speech data in the target language.

[0036] Specifically, the target speech recognition model can be understood as a speech recognition model specially designed for the target language, that is, the target speech recognition model is used to recognize language data in the target language. Among them, the target speech recognition model can be trained based on sample speech data in the target language, so that the target speech recognition model can learn the mapping relationship between sound features, vocabulary, grammatical structure and intonation in the target language and speech content, and then after receiving the wake-up speech data after the wake-up word is cut off, the target speech recognition model can accurately convert it into the corresponding text content, reduce misjudgment and ambiguity in the recognition process, and improve the recognition accuracy.

[0037] In addition, after performing voice recognition on the wake-up voice data after the wake-up word is cut off, the wake-up intention is determined based on the voice wake-up recognition result, and the wake-up operation corresponding to the wake-up intention is performed. For example, if the user's wake-up intention is "play music", the system will start the music playback function.

[0038] The voice wake-up recognition method provided in the embodiment of the present invention first recognizes the wake-up word for the short, clear and easy-to-recognize wake-up voice data to determine the starting point of the wake-up intention, which can not only reduce the amount of calculation, but also accurately identify multiple different wake-up words. In the case where there is a wake-up word in the wake-up voice data, the target language to which the wake-up word in the wake-up voice data belongs is determined, thereby ensuring that the target language recognition model matches the actual language requirement of the wake-up voice data, thereby improving the voice recognition accuracy. At the same time, the embodiment of the present invention cuts out the wake-up word in the wake-up voice data, thereby avoiding the interference of the wake-up word in the subsequent recognition of the user's wake-up intention, and further improving the voice wake-up recognition accuracy.

[0039] Based on the above embodiment, determining the target language of the wake-up word in the wake-up voice data includes: Extracting speech features of wake-up speech data; Encode the speech features to obtain coded features; Based on the coding features, determine the target language to which the wake-up word in the wake-up voice data belongs and the audio endpoint of the wake-up word.

[0040] Specifically, speech features can also be understood as acoustic features of wake-up speech data, which are used to characterize key information in wake-up speech data. Speech features can include Mel-scale Frequency Cepstral Coefficients (MFCC), linear predictive coding (LPC) and cepstral coefficients. The MFCC extraction process includes preprocessing, fast Fourier transform, Mei filter group, logarithmic operation, discrete cosine transform, dynamic feature extraction and other steps, such as: let the audio signal pass through a high-pass filter for pre-emphasis and compensation of high-frequency components; then divide the frame, generally take 25ms as a frame, and substitute each frame into the windowing function to smooth the discontinuity at both ends of a single signal frame; the windowed frame is subjected to fast Fourier transform to obtain the spectrum parameters of each frame; the spectrum parameters of each frame pass through the Mel filter group, take the logarithm of the output of each frequency band, and perform inverse transformation to obtain the Mel cepstral coefficient.

[0041] Considering that speech features usually contain a large number of data points, direct calculation based on speech features will consume a lot of computing resources and time. In addition, speech features may contain a lot of noise and redundant information, which will interfere with the accuracy of language recognition and audio endpoint detection.

[0042] In this regard, after extracting the voice features of the wake-up voice data, the embodiment of the present invention encodes the voice features, which can not only reduce the feature dimension and reduce the amount of calculation, but also extract key information and suppress noise, thereby ensuring the accuracy of language recognition and audio endpoint detection.

[0043] After obtaining the coding features, the coding features can be used to determine the target language of the wake-up word in the wake-up speech data and the audio endpoint of the wake-up word. For language recognition tasks, a language classifier can be trained and the coding features can be input into the classifier for language judgment. For audio endpoint detection tasks, the timing information and speech characteristics in the coding features can be used to accurately determine the starting and ending points of the effective speech part. Among them, the audio endpoint refers to the starting and ending points of the effective speech part in the speech signal.

[0044] Based on any of the above embodiments, performing wake-up word recognition on the wake-up voice data, and determining the target language to which the wake-up word in the wake-up voice data belongs when the wake-up word exists in the wake-up voice data, includes: Based on the wake-up word recognition model, the wake-up word recognition is performed on the wake-up voice data. If the wake-up word exists in the wake-up voice data, the target language of the wake-up word in the wake-up voice data and the audio endpoint of the wake-up word are determined; The wake-up word recognition model is obtained by fine-tuning the pre-trained wake-up model based on the first sample voice data in different languages, the language label of the sample wake-up word in each first sample voice data, and the audio endpoint label of the sample wake-up word. The pre-trained wake-up model is trained based on the second sample voice data in different languages ​​and the voice text labels of each second sample voice data.

[0045] Specifically, considering that directly training a new wake-up word recognition model requires a large amount of annotated data and computing resources, the cost is relatively high. In this regard, the embodiment of the present invention is fine-tuned based on the pre-trained wake-up model, and can use the general language knowledge that the pre-trained wake-up model has learned to reduce dependence on new data and reduce training costs. In addition, the pre-trained wake-up model is trained based on the second sample speech data of different languages ​​and the speech text labels of each second sample speech data, so that the pre-trained wake-up model has cross-language understanding capabilities, that is, the pre-trained wake-up model can recognize the commonalities and differences of speech in different languages, so as to better adapt to data in different languages ​​in subsequent tasks. Among them, the pre-trained wake-up model can be constructed based on RNN-T (RNN Transducer).

[0046] On this basis, the wake-up word recognition model is obtained by fine-tuning the pre-trained wake-up model based on the first sample voice data in different languages, the language label of the sample wake-up words in each first sample voice data, and the audio endpoint label of the sample wake-up words. It not only inherits the cross-language understanding ability of the pre-trained wake-up model, but also has additional capabilities such as accurate wake-up word recognition, language recognition and distinction, audio endpoint detection, stronger generalization ability and adaptive learning ability.

[0047] On the basis that the wake-up recognition model has the above-mentioned capabilities, based on the wake-up word recognition model, the wake-up voice data can be accurately recognized for the wake-up word, and when the wake-up word exists in the wake-up voice data, the target language to which the wake-up word belongs in the wake-up voice data and the audio endpoint of the wake-up word can be accurately determined. Among them, the target language may include Chinese, English, etc. Chinese can be subdivided into different types of dialects, such as a certain type of dialect, fuzzy dialect, etc. Among them, the above-mentioned second sample voice data may include the wake-up word, or may not include the wake-up word, and the embodiment of the present invention does not specifically limit this.

[0048] Based on any of the above embodiments, based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is cut off, including: Based on the target language of the wake-up word in the wake-up speech data, a target speech recognition model is determined from multiple speech recognition models, each speech recognition model is used to recognize speech data in a different language; Based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is removed.

[0049] Specifically, considering the significant differences in speech features, vocabulary, and grammatical structures between different languages, it is necessary to use a speech recognition model in the target language to identify the wake-up speech data. In this regard, an embodiment of the present invention determines a target speech recognition model from multiple speech recognition models based on the target language to which the wake-up word in the wake-up speech data belongs, and the target speech recognition model is used to identify language data in the target language. Among them, each speech recognition model is used to recognize speech data in different languages ​​to adapt to speech recognition in different languages.

[0050] For example, speech recognition model 1 can be used to recognize dialect 1, speech recognition model 2 can be used to recognize dialect 2, and speech recognition model 3 can be used to recognize English. It is understandable that speech recognition models of different languages ​​can be configured according to actual needs, and each speech recognition model is trained based on speech data of its own matching language, and the training method can be obtained by referring to the training method of the above-mentioned target speech recognition model.

[0051] Based on any of the above embodiments, the target speech recognition model includes an encoding module, a decoding module, and an attention module of the target speech recognition model; The target speech recognition model is trained based on the following steps: Based on the encoding module, encode the voice features of the third sample voice data to obtain sample encoding features, where the third sample voice data is the voice data after the wake-up word is cut off; Based on the attention module of the target speech recognition model, the attention calculation is performed on the sample encoding features to obtain the sample attention features; Based on the decoding module, the sample attention features are decoded to obtain the sample recognition text; Based on the difference between the sample recognition text and the speech text label corresponding to the third sample speech data, the initial model is trained to obtain a target speech recognition model.

[0052] Specifically, the speech feature of the third sample speech data can be understood as the acoustic feature of the third sample speech data, which is used to characterize the key information in the third sample speech data. Considering that the speech feature of the third sample speech data usually contains a large number of data points, if the calculation is performed directly based on the speech feature, a large amount of computing resources and time will be consumed. In addition, the speech feature may contain a large amount of noise and redundant information, which will interfere with the accuracy of language recognition and audio endpoint detection.

[0053] In this regard, the embodiment of the present invention encodes the speech features after extracting the speech features of the third sample speech data, which can not only reduce the feature dimension and the amount of calculation, but also extract key information and suppress noise, thereby ensuring the accuracy of language recognition and audio endpoint detection.

[0054] After obtaining the sample coding features, the sample coding features are subjected to attention calculation based on the attention module of the target speech recognition model to obtain the sample attention features. The attention calculation refers to integrating the key information into the sample attention features by calculating the attention weights of each position in the sample coding features to highlight the key information. After obtaining the sample attention features, the sample attention features are decoded by the decoding module to obtain the sample recognition text. Finally, the initial model is trained based on the difference between the sample recognition text and the speech text label (i.e., the real text) corresponding to the third sample speech data. During the training process, the initial model parameters are continuously adjusted to minimize the difference between the sample recognition text and the speech text label.

[0055] The embodiment of the present invention extracts sample coding features from the third sample voice data through the encoding module, performs weighted processing on the sample coding features by using the attention mechanism through the attention module to highlight key information, and converts the sample attention features into sample recognition text through the decoding module. That is, the embodiment of the present invention can directly and accurately convert voice data into text form through the collaborative work of the encoding module, the attention module and the decoding module.

[0056] Based on any of the above embodiments, each speech recognition model shares the encoding module and the decoding module, and during the training process of any speech recognition model, the attention module parameters of other speech recognition models are frozen.

[0057] Specifically, if each speech recognition model uses an independent encoding module and decoding module, each speech recognition module needs to be trained separately, resulting in a large amount of repeated training work, wasting computing resources and time. In this regard, in the embodiment of the present invention, the encoding module and the decoding module are shared among multiple speech recognition models, that is, the encoding module and the decoding module only need to be trained once and can be used by all speech recognition models, reducing the cost of repeated training and improving the overall training efficiency. In addition, the shared encoding module and the decoding module also promote knowledge transfer, so that different speech recognition models can use the same basic feature extraction and conversion capabilities, thereby speeding up training and improving model performance.

[0058] In addition, considering that the main task of the encoding module is to convert the input speech features into encoding features, and the decoding module is to convert the attention features into text form, they are universal for speech data of different languages, that is, the differences between the encoding module and the decoding module in different languages ​​are relatively small, and they can be shared without affecting the performance of the model. However, the main function of the attention module is to help the speech recognition model dynamically adjust its focus when processing the encoding features, so as to more effectively extract key features. Speech data of different languages ​​have differences in pronunciation, intonation, speaking speed, etc., and the encoding features of different languages ​​will also be different. Therefore, the attention module needs to be customized and optimized according to the characteristics of different languages ​​to ensure that the corresponding speech recognition model can accurately capture and process key information. Therefore, the encoding module and the decoding module can be shared because of their high versatility and computational efficiency, but the attention module cannot be shared because of its language specificity and important impact on model performance.

[0059] In addition, during the training process of any speech recognition model, if the attention module parameters of other speech recognition models are not frozen, the speech recognition models may interfere with each other. For example, the attention strategy of one model may affect the training process of another model, resulting in a decrease in model performance.

[0060] Based on this, when training any speech recognition model, the embodiment of the present invention shares the encoding module and the decoding module with other speech recognition models, and freezes the attention module parameters of other speech recognition models, which can not only significantly improve the training efficiency, but also avoid mutual interference between models and ensure model performance.

[0061] Figure 2 is a structural diagram of the wake-up word recognition model provided by the present invention, such as Figure 2 As shown, the wake-up word recognition model includes a first feature extraction module, a first encoding module, a language identification module, a first decoding module and an audio endpoint prediction module. Figure 3 It is a structural diagram of the speech recognition model provided by the present invention, such as Figure 3 As shown, the speech recognition model includes a second feature extraction module, a second encoding module, multiple language branches (such as a multi-language mixed branch, a language 1 branch, a language 2 branch, ..., a language n branch) and a second decoding module, wherein multiple dialect branches share the second encoding module and the second decoding module.

[0062] The wake-up word recognition model is trained based on the following steps: The second sample speech data of different languages ​​(such as different dialects) are input into the first feature extraction module to obtain the second speech features of each second sample speech data, and the second speech features are input into the first encoding module for feature encoding to obtain second sample encoding features, and the second sample encoding features are input into the first decoding module to predict and obtain sample recognition text, and based on the difference between the sample recognition text and the speech text label of the second sample speech data, the parameters of the first feature extraction module, the first encoding module and the first decoding module are updated to obtain a pre-trained wake-up model.

[0063] Next, the first sample voice data in different languages ​​containing the wake-up word is input into the first feature extraction module to obtain the first voice feature of the first sample voice data, and the first voice feature is input into the first encoding module for feature encoding to obtain the first sample encoding feature.

[0064] The first sample coding feature is input into the first decoding module, and the first decoding module recognizes the wake-up word in the first sample voice data; the first sample coding feature is input into the language discrimination module to obtain the target language to which the wake-up word belongs; the first sample coding feature is input into the audio endpoint prediction module to obtain the audio endpoint of the wake-up word.

[0065] Based on the difference between the wake-up word and the wake-up word label, the difference between the target language and the language label, and the difference between the audio endpoint and the audio endpoint label, the parameters of the first feature extraction module, the first encoding module, the first decoding module, the language discrimination module and the audio endpoint prediction module are adjusted to obtain a wake-up recognition model.

[0066] The speech recognition model is trained based on the following steps: Based on the fourth sample speech data in different languages ​​and the corresponding speech text labels, the initial model of the speech recognition model is trained to obtain a pre-trained speech model.

[0067] Using the wake-up recognition model trained as above, the wake-up words in the third sample voice data of different languages, the target language to which the wake-up words belong, and the audio endpoint of the wake-up words in the third sample voice data are identified, and the wake-up words are cut out from the third sample voice data based on the audio endpoint.

[0068] The third sample speech data after the wake-up word is cut off is input into the second feature extraction module to obtain the third sample speech feature; the third sample speech feature is input into the second encoding module to obtain the third sample encoding feature; the third sample encoding feature is input into the target branch, and the target branch performs attention calculation on the third sample encoding feature to obtain the sample attention feature; the sample attention feature is input into the second decoding module to obtain the sample recognition text. Among them, the target branch is determined based on the target language to which the wake-up word belongs, and the target branch is used to perform attention calculation on the encoding feature under the target language. The language discrimination module in the wake-up word recognition model can output the value corresponding to the target language, and the recognition model determines the corresponding target branch based on the value.

[0069] Based on the difference between the sample recognition text and the corresponding speech text label, the parameters of the second feature extraction module, the second encoding module, the target branch and the second decoding module in the pre-trained speech model are updated. When the parameters of any language branch are updated, the parameters of other language branches are frozen. For example, when the parameters of the target branch are updated, the parameters of the remaining language branches are frozen. Finally, after the parameters of all language branches are updated, the speech recognition model is obtained.

[0070] The voice wake-up recognition device provided by the present invention is described below. The voice wake-up recognition device described below and the voice wake-up recognition method described above can be referred to each other.

[0071] Based on any of the above embodiments, Figure 4 is a schematic diagram of the structure of the voice wake-up recognition device provided by the present invention, such as Figure 4 As shown, the device comprises: The recognition unit 410 is used to recognize the wake-up word of the wake-up voice data; A determination unit 420, configured to determine the target language to which the wake-up word in the wake-up speech data belongs, and to remove the wake-up word from the wake-up speech data, if the wake-up word exists in the wake-up speech data; The wake-up unit 430 is used to perform speech recognition on the wake-up speech data after the wake-up word is cut out based on the target speech recognition model to obtain a speech wake-up recognition result. The target speech recognition model is used to recognize speech data in a target language.

[0072] Based on any of the foregoing embodiments, determining the target language to which the wake-up word in the wake-up voice data belongs includes: Extracting speech features of wake-up speech data; Encode the speech features to obtain coded features; Based on the encoding features, determine the target language of the wake-up word in the wake-up speech data and the audio endpoint of the wake-up word.

[0073] Based on any of the above embodiments, performing wake-up word recognition on the wake-up voice data, and determining the target language to which the wake-up word in the wake-up voice data belongs when the wake-up word exists in the wake-up voice data, includes: Based on the wake-up word recognition model, the wake-up word recognition is performed on the wake-up voice data. If the wake-up word exists in the wake-up voice data, the target language of the wake-up word in the wake-up voice data and the audio endpoint of the wake-up word are determined; The wake-up word recognition model is obtained by fine-tuning the pre-trained wake-up model based on the first sample voice data in different languages, the language label of the sample wake-up word in each first sample voice data, and the audio endpoint label of the sample wake-up word. The pre-trained wake-up model is trained based on the second sample voice data in different languages ​​and the voice text labels of each second sample voice data.

[0074] Based on any of the above embodiments, based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is cut off, including: Based on the target language of the wake-up word in the wake-up speech data, a target speech recognition model is determined from multiple speech recognition models, each speech recognition model is used to recognize speech data in a different language; Based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is removed.

[0075] Based on any of the above embodiments, the target speech recognition model includes an encoding module, a decoding module, and an attention module of the target speech recognition model; the target speech recognition model is trained based on the following steps: Based on the encoding module, encode the voice features of the third sample voice data to obtain sample encoding features, where the third sample voice data is the voice data after the wake-up word is cut off; Based on the attention module of the target speech recognition model, the attention calculation is performed on the sample encoding features to obtain the sample attention features; Based on the decoding module, the sample attention features are decoded to obtain the sample recognition text; Based on the difference between the sample recognition text and the speech text label corresponding to the third sample speech data, the initial model is trained to obtain a target speech recognition model.

[0076] Based on any of the above embodiments, each speech recognition model shares the encoding module and the decoding module, and during the training process of any speech recognition model, the attention module parameters of other speech recognition models are frozen.

[0077] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the voice wake-up recognition method, which includes: performing wake-up word recognition on the wake-up voice data; when the wake-up word exists in the wake-up voice data, determining the target language to which the wake-up word in the wake-up voice data belongs, and cutting the wake-up word from the wake-up voice data; based on the target voice recognition model, performing voice recognition on the wake-up voice data after cutting the wake-up word to obtain a voice wake-up recognition result, and the target voice recognition model is used to recognize the voice data in the target language.

[0078] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech wake-up recognition method provided by the above methods, which includes: performing wake-up word recognition on the wake-up speech data; when the wake-up word exists in the wake-up speech data, determining the target language to which the wake-up word in the wake-up speech data belongs, and cutting the wake-up word from the wake-up speech data; based on the target speech recognition model, performing speech recognition on the wake-up speech data after cutting off the wake-up word to obtain a speech wake-up recognition result, and the target speech recognition model is used to recognize speech data in the target language.

[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech wake-up recognition method provided by the above-mentioned methods, the method comprising: performing wake-up word recognition on the wake-up speech data; when the wake-up word exists in the wake-up speech data, determining the target language to which the wake-up word in the wake-up speech data belongs, and cutting the wake-up word from the wake-up speech data; based on a target speech recognition model, performing speech recognition on the wake-up speech data after cutting off the wake-up word to obtain a speech wake-up recognition result, the target speech recognition model being used to recognize speech data in the target language.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice wake-up recognition method, characterized in that: include: Perform wake-up word recognition on wake-up voice data; In the case where the wake-up word exists in the wake-up voice data, determining the target language to which the wake-up word in the wake-up voice data belongs, and removing the wake-up word from the wake-up voice data; Based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is cut off to obtain a speech wake-up recognition result, and the target speech recognition model is used to recognize the speech data in the target language.

2. The voice wake-up recognition method according to claim 1, characterized in that: The determining the target language to which the wake-up word in the wake-up voice data belongs includes: Extracting voice features of the wake-up voice data; Encoding the speech feature to obtain a coded feature; Based on the encoding feature, determine the target language to which the wake-up word in the wake-up voice data belongs and the audio endpoint of the wake-up word.

3. The voice wake-up recognition method according to claim 2, characterized in that: The performing wake-up word recognition on the wake-up voice data, and determining a target language to which the wake-up word in the wake-up voice data belongs when the wake-up word exists in the wake-up voice data, includes: Based on the wake-up word recognition model, perform wake-up word recognition on the wake-up voice data, and if the wake-up word exists in the wake-up voice data, determine the target language to which the wake-up word in the wake-up voice data belongs and the audio endpoint of the wake-up word; The wake-up word recognition model is obtained by fine-tuning a pre-trained wake-up model based on first sample voice data in different languages, the language label of the sample wake-up word in each first sample voice data, and the audio endpoint label of the sample wake-up word. The pre-trained wake-up model is trained based on second sample voice data in different languages ​​and the voice text label of each second sample voice data.

4. The voice wake-up recognition method according to any one of claims 1 to 3, characterized in that: The performing speech recognition on the wake-up speech data after the wake-up word is cut out based on the target speech recognition model includes: Based on the target language to which the wake-up word in the wake-up voice data belongs, determining a target voice recognition model from a plurality of voice recognition models, each voice recognition model being used to recognize voice data in a different language; Based on the target speech recognition model, speech recognition is performed on the wake-up speech data after the wake-up word is cut out.

5. The voice wake-up recognition method according to claim 4, characterized in that: The target speech recognition model includes an encoding module, a decoding module and an attention module of the target speech recognition model; the target speech recognition model is trained based on the following steps: Based on the encoding module, encoding the voice features of the third sample voice data to obtain a sample encoding feature, wherein the third sample voice data is the voice data after the wake-up word is cut off; Based on the attention module of the target speech recognition model, performing attention calculation on the sample coding feature to obtain a sample attention feature; Based on the decoding module, the sample attention feature is decoded to obtain a sample recognition text; Based on the difference between the sample recognition text and the speech text label corresponding to the third sample speech data, the initial model is trained to obtain the target speech recognition model.

6. The voice wake-up recognition method according to claim 5, characterized in that: Each speech recognition model shares the encoding module and the decoding module, and during the training process of any speech recognition model, the attention module parameters of other speech recognition models are frozen.

7. A voice wake-up recognition device, characterized in that: include: A recognition unit, used to recognize the wake-up word of the wake-up voice data; a determining unit, configured to determine, when the wake-up word exists in the wake-up voice data, a target language to which the wake-up word in the wake-up voice data belongs, and to delete the wake-up word from the wake-up voice data; The wake-up unit is used to perform speech recognition on the wake-up speech data after the wake-up word is cut out based on a target speech recognition model to obtain a speech wake-up recognition result. The target speech recognition model is used to recognize the speech data in the target language.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the voice wake-up recognition method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice wake-up recognition method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the voice wake-up recognition method according to any one of claims 1 to 6 is implemented.

Images