Speech recognition method, terminal equipment and storage medium

Abstract

The invention discloses a voice recognition method, terminal equipment and a storage medium, and the method comprises the steps: carrying out the voiceprint feature extraction of a to-be-recognized voice, and determining the first voiceprint information corresponding to the to-be-recognized voice; inputting the to-be-recognized voice and the first voiceprint information into a voice recognition model to obtain a recognition result corresponding to the to-be-recognized voice; wherein the voice recognition model is obtained by training the self-attention network through the enhanced voice data.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method, a terminal device and a storage medium. Background Art

[0002] Speaker Age Estimation (SAE) is widely considered to be a sub-problem of speech attribute recognition, which estimates the speaker's age from the acquired audio data.

[0003] However, common speech recognition methods cannot accurately identify the speaker's age information in some audio scenarios where the scenes are complex and diverse and the audio signal-to-noise ratio is low. There is a problem of low speech recognition accuracy, which reduces the intelligence of the terminal device. Summary of the invention

[0004] The embodiments of the present application provide a speech recognition method, a terminal device, and a storage medium, which can improve the accuracy of speech recognition, thereby improving the intelligence of the terminal device.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a speech recognition method, the method comprising:

[0007] Extracting voiceprint features of the speech to be recognized, and determining first voiceprint information corresponding to the speech to be recognized;

[0008] The speech to be recognized and the first voiceprint information are input into a speech recognition model to obtain a recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training a self-attention network with enhanced speech data.

[0009] In a second aspect, an embodiment of the present application provides a terminal device, the terminal device comprising: a determining unit, an acquiring unit,

[0010] The determining unit is used to extract voiceprint features of the speech to be recognized and determine first voiceprint information corresponding to the speech to be recognized;

[0011] The acquisition unit is used to input the speech to be recognized and the first voiceprint information into a speech recognition model to obtain a recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training a self-attention network with enhanced speech data.

[0012] In a third aspect, an embodiment of the present application provides a terminal device, which includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method described in the first aspect above is implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the method described in the first aspect above is implemented.

[0014] The embodiment of the present application provides a speech recognition method, a terminal device and a storage medium, wherein the terminal device extracts voiceprint features of the speech to be recognized, determines the first voiceprint information corresponding to the speech to be recognized, inputs the speech to be recognized and the first voiceprint information into the speech recognition model, and obtains the recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training the self-attention network with the enhanced speech data. That is to say, in the embodiment of the present application, the terminal device integrates the voiceprint features of the speech in the process of speech recognition, and improves the accuracy of speech recognition. At the same time, the speech recognition model used by the terminal device for speech recognition processing is trained with the enhanced speech data, which can effectively improve the robustness of the speech recognition model, so that the speech recognition model can be better applied to complex audio scenes, and the speech recognition model can make full use of the contextual information of the speech through the self-attention network, further improving the accuracy of speech recognition. It can be seen that in the embodiment of the present application, based on the combined use of voiceprint features and speech recognition models, the age information of the speaker can be accurately identified, which greatly improves the accuracy of speech recognition, thereby improving the intelligence of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the implementation process of the speech recognition method Figure 1 ;

[0016] Figure 2 Schematic diagram of the voiceprint recognition network structure Figure 1 ;

[0017] Figure 3 Schematic diagram of the voiceprint recognition network structure Figure 2 ;

[0018] Figure 4 This is a schematic diagram of the structure of the speech recognition model. Figure 1 ;

[0019] Figure 5 This is a schematic diagram of the structure of the speech recognition model. Figure 2 ;

[0020] Figure 6 Schematic diagram of the point-by-point attention mechanism network;

[0021] Figure 7 Schematic diagram of a multi-head self-attention mechanism network;

[0022] Figure 8 It is a schematic diagram of the use process of the speech recognition model;

[0023] Fig. 9 Schematic diagram of the training process of the speech recognition model;

[0024] Fig.10 This is a schematic diagram of an application scenario of a speech recognition solution;

[0025] Fig.11 This is a framework diagram for implementing the speech recognition method;

[0026] Fig.12 A schematic diagram of the structure of the speech recognition model proposed in the embodiment of the present application;

[0027] Fig.13 Schematic diagram of the structure of the terminal device proposed in the embodiment of the present application Figure 1 ;

[0028] Fig.14 Schematic diagram of the structure of the terminal device proposed in the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for ease of description, only the parts related to the related applications are shown in the drawings.

[0030] With the development of mobile terminal technology, consumers' demand for mobile terminal photo albums is also moving towards intelligentization. Therefore, how to make full use of the voice information in the album video to make the album editing technology more intelligent is a direction worth studying.

[0031] Speaker Age Estimation (SAE) is widely considered to be a sub-problem of speech attribute recognition, which estimates the speaker's age (child, middle-aged, elderly) through the acquired audio data.

[0032] Common SAE technologies are mainly based on traditional machine learning (non-deep learning) solutions, such as using the Support Vector Machines (SVM) method to estimate the speaker's age; the ivert method in Speaker Recognition (SR) is also a commonly used technology. In addition, since there are certain differences in parameters such as vocal cord coefficients and resonance peak frequencies between speakers of different ages, the acoustic features can also be used for classification. These solutions require less training data and have fast recognition speeds, and have good recognition accuracy for scenarios with high signal-to-noise ratios.

[0033] However, in the scenario of user album videos, the existence of a large number of low signal-to-noise ratio scenes will cause the accuracy of age estimation to drop seriously. Therefore, the above method cannot be applied to create wonderful childhood memories of user videos.

[0034] With the development of deep learning technology, in the field of SAE, methods based on deep neural networks (DNN) have shown performance superior to traditional machine learning methods. The current mainstream SAE system uses speech acoustic features (such as FBANK (Filter Bank) features) as input, and then trains the neural network model, which generally achieves good results in relatively matching test scenarios, such as using generative adversarial networks (GAN) for age recognition, or using residual networks (ResNet) for speaker age estimation.

[0035] However, the following problems are still difficult to solve in user album videos: 1. User video scenes vary widely, and there may be user videos with very low signal-to-noise ratio, which places high demands on the robustness of the SAE model; 2. Existing technologies directly extract acoustic characteristics of speech and then train neural network models, ignoring the voiceprint characteristics of speech, while voiceprint information is helpful for SAE estimation; 3. Existing SAE based on deep neural network technology finds it difficult to model the temporal information of a piece of speech well, which also leads to a decrease in the accuracy of age estimation.

[0036] In other words, the common traditional machine learning or deep learning technology SAE solution cannot be effectively applied in scenarios such as terminal albums. This is because in the user album videos, the videos have the characteristics of complex and diverse scenes and low audio signal-to-noise ratio. Therefore, for SAE technology, it is necessary to have strong robustness, be able to cope with various scenarios, and be able to cope with low signal-to-noise ratio. In addition, in the technical framework of creating wonderful memories of children's voices, the SAE module determines whether the voice clips are children's voices, so it needs to have extremely high accuracy. However, the current mainstream SAE technology is difficult to meet the requirements for age estimation accuracy in creating wonderful memories of children's voices.

[0037] It can be seen that the traditional SAE method based on machine learning has a serious drop in recognition rate in low signal-to-noise ratio scenarios, and it is difficult to achieve good results in user album video scenarios. The current mainstream SAE method based on deep learning cannot obtain accurate age estimation results in user album video scenarios, where voice signal scenarios vary widely and there are user videos with very low signal-to-noise ratios.

[0038] In other words, common speech recognition methods cannot accurately identify the speaker's age information in some audio scenarios where the scenes are complex and diverse and the audio signal-to-noise ratio is low. There is a problem of low speech recognition accuracy, which reduces the intelligence of the terminal device.

[0039] In order to solve the above problems, in an embodiment of the present application, the terminal device extracts voiceprint features of the speech to be recognized, determines the first voiceprint information corresponding to the speech to be recognized; inputs the speech to be recognized and the first voiceprint information into the speech recognition model, and obtains the recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training the self-attention network with the enhanced speech data. That is to say, in an embodiment of the present application, the terminal device integrates the voiceprint features of the speech in the process of speech recognition, and improves the accuracy of speech recognition. At the same time, the speech recognition model used by the terminal device for speech recognition processing is trained with the enhanced speech data, which can effectively improve the robustness of the speech recognition model, so that the speech recognition model can be better applied to complex audio scenes, and the speech recognition model can make full use of the contextual information of the speech through the self-attention network, further improving the accuracy of speech recognition. It can be seen that in an embodiment of the present application, based on the combined use of voiceprint features and speech recognition models, the age information of the speaker can be accurately identified, which greatly improves the accuracy of speech recognition, thereby improving the intelligence of the terminal device.

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0041] An embodiment of the present application provides a speech recognition method, which can be applied to a terminal device. Figure 1 Schematic diagram of the implementation process of the speech recognition method Figure 1 ,like Figure 1 As shown, the method for performing speech recognition by a terminal device may include the following steps:

[0042] Step 101: extract voiceprint features of the speech to be recognized, and determine first voiceprint information corresponding to the speech to be recognized.

[0043] In an embodiment of the present application, the terminal device may first extract voiceprint features of the speech to be recognized, so as to determine the first voiceprint information corresponding to the speech to be recognized.

[0044] It should be noted that, in the embodiments of the present application, the speech to be recognized may be speech information in any format, and the present application does not make any specific limitation.

[0045] It should be noted that in the embodiments of the present application, the terminal device can acquire the speech to be recognized in any manner, and the present application does not make any specific limitation.

[0046] Exemplarily, in some embodiments, the voice to be recognized may be voice information extracted by the terminal device from the video, or may be voice information acquired by the terminal device through a voice acquisition device, or may be voice information sent by other devices and received by the terminal device.

[0047] Furthermore, in an embodiment of the application, the terminal device may use a voiceprint recognition network to perform voiceprint recognition processing on the speech to be recognized, thereby determining the first voiceprint information corresponding to the speech to be recognized.

[0048] It should be noted that, in the embodiment of the present application, the voiceprint recognition network is used to extract the voiceprint features of the speech to be recognized, and finally obtain the corresponding first voiceprint information for characterizing the voiceprint features.

[0049] It can be understood that in the embodiments of the present application, the voiceprint recognition network can be understood as a voiceprint recognition system based on a neural network, and the voiceprint recognition network can be a ResNet network structure, wherein the voiceprint recognition network includes a frame-level feature learning layer, a feature aggregation layer, and a sentence-level feature learning layer.

[0050] It should be noted that, in the embodiments of the present application, the first voiceprint information obtained based on the voiceprint recognition network may be a feature vector that can characterize the voiceprint features of the speech to be recognized. For example, the first voiceprint information may be an r-vector vector.

[0051] Exemplarily, in some embodiments, Figure 2 Schematic diagram of the voiceprint recognition network structure Figure 1 ,like Figure 2 As shown in the figure, the voiceprint recognition network can adopt a network structure based on ResNet, and the specific structure can be divided into a frame-level deep feature learning layer (Frame-level deep feature learning, i.e., frame-level feature learning layer), a feature aggregation layer (Aggregation), and a sentence-level representation learning layer (Segment-leval embedding learning, i.e., sentence-level feature learning layer). Among them, the voiceprint recognition network model is trained through the speaker distinguishing training criterion, and the speaker representation (Speaker embedding) vector is extracted at the sentence level layer. The final extracted speaker representation vector is referred to as r-vector, i.e., voiceprint information.

[0052] Exemplarily, in some embodiments, Figure 3 Schematic diagram of the voiceprint recognition network structure Figure 2 ,like Figure 3 As shown in the figure, the voiceprint recognition network adopts a 34-layer ResNet network structure as an example, in which the first part is a layer of Conv2d, which increases the number of channels of input features while ensuring that the time dimension and frequency dimension remain unchanged; the second part is a stack of 32 layers of Conv2d (ResNetBlock-1, ResNetBlock-2, ResNetBlock-3, ResNetBlock-4), which convolve the time dimension and frequency dimension to extract the high-dimensional features of the speech; the third part is the StatsPooling layer and the Flatten layer, which are used to extract the mean and variance of the sentence level, so that the feature representation of the frame level is converted to the feature representation of the sentence level; the fourth part is the fully connected neural network Dense1 and Dense2, in which Dense1 reduces the feature representation of the sentence level to 256 dimensions, which is the vector representation r-vector of the voiceprint feature. Dense2 maps r-vector to the final number of speakers N, and uses the softmax function to obtain the posterior probability representation of the speaker classification.

[0053] Step 102: input the to-be-recognized speech and the first voiceprint information into a speech recognition model to obtain a recognition result corresponding to the to-be-recognized speech; wherein the speech recognition model is obtained by training a self-attention network with enhanced speech data.

[0054] In an embodiment of the present application, after the terminal device extracts voiceprint features of the speech to be recognized and determines the first voiceprint information corresponding to the speech to be recognized, it can further input the speech to be recognized and the first voiceprint information into the speech recognition model, so as to obtain the recognition result corresponding to the speech to be recognized.

[0055] It should be noted that, in the embodiments of the present application, the speech recognition model can be obtained by training the Self-Attention Based Network (SAN) through enhanced speech data. Among them, the enhanced speech data can make the trained speech recognition model more robust, and the self-attention network has a strong ability to model the temporal information of speech, which can improve the speech recognition accuracy of the speech recognition model.

[0056] That is to say, in an embodiment of the present application, the terminal device can use the enhanced voice data to perform model training on the initial recognition model composed of the self-attention network, and finally obtain the corresponding voice recognition model.

[0057] Furthermore, in an embodiment of the present application, the speech recognition model obtained through training may include a first self-attention network and a second self-attention network. Accordingly, when the terminal device inputs the speech to be recognized and the first voiceprint information into the speech recognition model to obtain the recognition result corresponding to the speech to be recognized, it can first extract attribute features of the speech to be recognized through the first self-attention network to obtain the first attribute information corresponding to the speech to be recognized; then, based on the first attribute information and the first voiceprint information, it can determine the fused feature information corresponding to the speech to be recognized; finally, through the second self-attention network and the fused feature information, it can determine the recognition result corresponding to the speech to be recognized.

[0058] That is to say, in the embodiment of the present application, the first self-attention network included in the speech recognition model can be used to extract the attribute features corresponding to the speech to be recognized. Among them, the first attribute information corresponding to the speech to be recognized can determine the age attribute corresponding to the speech to be recognized.

[0059] It should be noted that, in an embodiment of the present application, the first attribute information obtained based on the first self-attention network may be a feature vector that can characterize the age characteristics of the speech to be recognized. For example, the first attribute information may be an a-vector vector.

[0060] It can be understood that in the embodiments of the present application, the second self-attention network included in the speech recognition model can be used to perform further feature extraction processing on the attribute information of the speech to be recognized after the first attribute information and the first voiceprint information are fused.

[0061] It should be noted that, in the embodiments of the present application, the information input into the speech recognition model, in addition to the first attribute information characterizing the age characteristics of the speech to be recognized, also includes the first voiceprint information characterizing the voiceprint characteristics of the speech to be recognized. The voiceprint feature information is integrated into the speech recognition model, so that the speech recognition model can learn the voiceprint features in the speech at the same time, which can improve the accuracy of speech recognition.

[0062] Exemplarily, in some embodiments, Figure 4 This is a schematic diagram of the structure of the speech recognition model. Figure 1 ,like Figure 4 As shown, the speech recognition model includes two self-attention networks, such as the first self-attention network Self-Attention1 and the second self-attention network Self-Attention2, wherein Self-Attention1 can extract the age attribute information a-vector vector in the speech to be recognized, that is, obtain the first attribute information, and then add it to the first voiceprint information r-vector vector representing the voiceprint feature to obtain the fused feature information; then the second self-attention network Self-Attention2 is used to learn the fused feature information obtained after the fusion of a-vector and r-vector, and finally obtain the recognition result corresponding to the speech to be recognized.

[0063] Furthermore, in an embodiment of the present application, the speech recognition model obtained through training may include a fully connected layer. Accordingly, when the terminal device determines the recognition result corresponding to the speech to be recognized through the second self-attention network and the fused feature information, it can first learn the fused feature information through the second self-attention network to obtain the feature information corresponding to the speech to be recognized; then, the feature information corresponding to the speech to be recognized can be classified through the fully connected layer to determine the recognition result corresponding to the speech to be recognized.

[0064] It can be understood that in the embodiments of the present application, the fully connected layer included in the speech recognition model can perform classification processing based on the learned feature information, that is, the fully connected layer can further map the feature information corresponding to the speech to be recognized to the labeled sample space, thereby obtaining the final recognition result.

[0065] Exemplarily, in some embodiments, Figure 5 This is a schematic diagram of the structure of the speech recognition model. Figure 2 ,like Figure 5As shown, the speech recognition model also includes a fully connected layer (FC). For the feature information of the speech to be recognized output by the second self-attention network Self-Attention2 after learning the fused feature information, the fully connected layer can map it to the output node (such as age category), that is, obtain the recognition result corresponding to the speech to be recognized.

[0066] It should be noted that, in an embodiment of the present application, the first self-attention network and the second self-attention network both include multiple layers of self-attention layers; wherein the self-attention layers include a multi-head self-attention mechanism network and a feedforward network.

[0067] It is understandable that in the embodiments of the present application, the network structures of the first attention network and the second attention network can be the same, wherein the attention networks can be composed of multiple layers of self-attention layers. Each self-attention layer can include two sub-layers, namely a multi-head self-attention mechanism network and a feedforward network.

[0068] It should be noted that, in the embodiments of the present application, the multi-head self-attention mechanism network is a multi-head self-attention mechanism (Multi-Head Attention, MHA), and the feedforward network is a simple feedforward network (Feed Forward Network, FFN).

[0069] It should be noted that in the embodiments of the present application, for each sub-layer, a residual connection (RC) can be selected, and then layer normalization (LN) can be performed. In order to be consistent with the dimension of the voiceprint feature r-vector vector, all sub-layers in the model generate outputs with the same dimension as the r-vector vector. For example, if the dimension of the voiceprint feature r-vector vector is 256, then each sub-layer can also generate an output with a dimension of 256.

[0070] Further, in an embodiment of the present application, the multi-head self-attention mechanism can be described as mapping a query (Query, Q) and a set of key-value pairs (Key-Value) to an output, where the query (Q), key (K), value (V) and output are all vectors. The output is calculated by the weighted sum of the values, where the weight assigned to each value is calculated by the value of Q and the corresponding K. The core idea of ​​MHA is to project Q, K, and V through h different linear transformations, and finally splice the different attention results together.

[0071] Exemplarily, in some embodiments, Figure 6This is a schematic diagram of the point-by-point attention mechanism network, as shown Figure 6 As shown in the figure, for the Scaled Dot-Product Attention (SDPA) mechanism, a set of queries is packed into the matrix Q, and the keys and values ​​are also packed into the matrices K and V respectively. The output matrix calculation formula is:

[0072]

[0073] Among them, d k is the scaling factor, for example, d k Set to 32.

[0074] Exemplarily, in some embodiments, Figure 7 This is a schematic diagram of a multi-head self-attention mechanism network. Figure 7 As shown in the figure, in contrast, the FFN layer in the Self-Attention layer is essentially a two-layer fully connected neural network. The activation function of the first layer is generally a linear rectified function (Rectified Linear Unit, ReLU), and the second layer is a linear activation function. If the output of the MHA layer is represented as Z, then the FFN can be expressed as:

[0075] FFN(Z)=max(0,ZW 1 +b 1 )W 2 +b 2 (2)

[0076] It should be noted that in an embodiment of the present application, when determining the fused feature information corresponding to the speech to be recognized based on the first attribute information and the first voiceprint information, the terminal device can fuse the first attribute information and the first voiceprint information according to a preset fusion strategy, thereby obtaining the fused feature information.

[0077] It is understandable that in the embodiments of the present application, the preset fusion strategy can be used to fuse different information. Based on the preset fusion strategy, the terminal device can select any fusion algorithm to fuse the first attribute information and the first voiceprint information. For example, the preset fusion strategy includes addition fusion, or point multiplication fusion, or splicing fusion. This application does not make specific limitations.

[0078] Exemplarily, in some embodiments, the terminal device may use an additive fusion algorithm to fuse the first attribute information and the first voiceprint information; or, the terminal device may use a dot product fusion algorithm to fuse the first attribute information and the first voiceprint information; or, the terminal device may use a splicing fusion algorithm to fuse the first attribute information and the first voiceprint information.

[0079] It should be noted that, in the embodiments of the present application, the speech recognition model can be used to predict the age corresponding to the speech to be recognized, so the recognition result corresponding to the speech to be recognized can include age-related information. For example, the recognition result of the speech to be recognized can be a specific age value range, or an age type classified based on age. The present application does not specifically limit this.

[0080] Exemplarily, in some embodiments, the recognition result corresponding to the voice to be recognized determined by the terminal based on the voice recognition model can be under 10 years old, or 10 to 16 years old, or 16 to 25 years old, or 25 to 45 years old, or over 45 years old.

[0081] Exemplarily, in some embodiments, the recognition result corresponding to the speech to be recognized determined by the terminal based on the speech recognition model may be a child, or a middle-aged person, or an elderly person.

[0082] Exemplarily, in some embodiments, Figure 8 This is a schematic diagram of the use process of the speech recognition model. Figure 8 As shown in the figure, the process of using the model is mainly based on two parts, including the SAE model (i.e., speech recognition model) and the voiceprint recognition network. For the input speech to be recognized, on the one hand, the voiceprint feature of the speech to be recognized can be extracted through the voiceprint recognition network to obtain the corresponding voiceprint feature r-vector (i.e., the first voiceprint information); on the other hand, the speech to be recognized can be input into the first self-attention network in the SAE model, and the corresponding age attribute feature a-vector (i.e., the first attribute information) can be obtained through the first self-attention network. Then, the age attribute feature and the voiceprint feature can be fused, and the fused information (i.e., the fused feature information) can be input into the second self-attention network for learning to obtain the corresponding feature information, and then the input feature information can be mapped to the output node through the fully connected layer to obtain the corresponding recognition result.

[0083] Furthermore, in the real time of the present application, during the training process of the speech recognition model, the terminal device needs to first perform data enhancement processing on the speech data in the speech data set so as to obtain enhanced speech data; then the enhanced speech data can be used to train the initial recognition model to obtain the speech recognition model.

[0084] It should be noted that in an embodiment of the present application, in order to make the speech recognition model more robust, data enhancement processing can be performed on the speech data in the original speech data set before model training, wherein the data enhancement processing can include at least one or more of speed change processing, noise addition processing, and reverberation processing.

[0085] It can be understood that in the embodiments of the present application, the speed change processing can be used to change the speech speed of the voice data according to the preset speech speed parameter; wherein the preset speech speed parameter can be any value greater than 0, and the present application does not make any specific limitation.

[0086] For example, in some embodiments, when the speech data is processed at a variable speed, the principle of variable speed without changing pitch can be adopted, and each speech data is respectively changed to 0.9, 1.0, and 1.1 times the original speech speed, while ensuring that the pitch remains unchanged. The variable speed without changing pitch processing can simulate the speech speed changes of the speaker in the actual scene, thereby improving the robustness of the model.

[0087] It can be understood that in the embodiments of the present application, the noise addition process can be used to add noise to the speech data according to the noise data; wherein the noise data can be obtained from any speech data set, and the present application does not make any specific limitation.

[0088] Exemplarily, in some embodiments, when adding noise to speech data, the open source MUSAN dataset can be used to add noise to speech data. The MUSAN dataset contains three types of noise: speech, music, and noise. For speech noise, the signal-to-noise ratio of the added noise is a random value of 0dB-15dB; for music noise, the signal-to-noise ratio of the added noise is a random value of 5dB-15dB; for noise noise, the signal-to-noise ratio of the added noise is a random value of 10dB-20dB.

[0089] It should be noted that, in the embodiments of the present application, the MUSAN data set covers many types of noise, and one type of noise can be randomly selected for each speech data for noise addition, and the noise addition process covers different signal-to-noise ratios. Therefore, it can simulate complex scenes in various types of videos to a certain extent, thereby improving the accuracy of the model.

[0090] It can be understood that, in the embodiments of the present application, the reverberation processing can be used to add reverberation to the speech data according to the reverberation data; wherein the reverberation data can be obtained from any speech data set, and the present application does not specifically limit it.

[0091] Exemplarily, in some embodiments, when performing reverberation processing on speech data, the simulated_rirs reverberation data in the open source RIRS data set can be used to perform reverberation processing on the speech signal. Specifically, a reverberation data can be randomly selected to perform reverberation processing on the input speech data.

[0092] It should be noted that, in the embodiments of the present application, the RIRS data set covers a large amount of reverberation data, and the reverberation processing can better simulate the real reverberation scene, thereby improving the accuracy of the model.

[0093] It can be understood that in an embodiment of the present application, the initial recognition model has the same model structure as the speech recognition model obtained after training, that is, the initial recognition model can include a first self-attention network, a second self-attention network and a fully connected layer.

[0094] Furthermore, in an embodiment of the present application, after completing the data enhancement processing of the speech data set, when using the enhanced speech data to train the initial recognition model to obtain the speech recognition model, the voiceprint recognition network can be used to perform voiceprint recognition processing on the speech data in the speech data set to determine the second voiceprint information corresponding to the enhanced speech data; then the first self-attention network can be used to extract attribute features of the enhanced speech data to obtain the second attribute information corresponding to the enhanced speech data; then the second self-attention network can be used to learn the fusion features to determine the feature information corresponding to the enhanced speech data; wherein the fusion features are obtained by fusing the second attribute information and the second voiceprint information; and then the feature information corresponding to the enhanced speech data can be classified through the fully connected layer to determine the training result corresponding to the enhanced speech data; finally, the loss function can be calculated based on the training result, and the initial recognition model can be corrected according to the loss function to obtain the speech recognition model.

[0095] Exemplarily, in some embodiments, Fig. 9 This is a diagram of the training process of the speech recognition model. Fig. 9 As shown in the figure, the model training process is mainly based on three parts, including data enhancement, SAE model (i.e., initial recognition model), and voiceprint recognition network. For the voice data in the input voice data set, on the one hand, the corresponding enhanced voice data can be obtained through data enhancement processing such as speed change processing, noise addition processing, and reverberation processing. On the other hand, the voiceprint feature of the voice data in the voice data set can be extracted through the voiceprint recognition network to obtain the corresponding voiceprint feature r-vector (i.e., the second voiceprint information); then, the enhanced voice data and voiceprint feature can be input into the SAE model. Specifically, the enhanced voice data can be input into the first self-attention network, and the corresponding age attribute feature a-vector (i.e., the second attribute information) can be obtained through the first self-attention network. Then, the age attribute feature and the voiceprint feature can be fused, and the fused information (i.e., fused feature) can be input into the second self-attention network for learning to obtain the corresponding feature information, and then the input feature information can be mapped to the output node through the fully connected layer to obtain the corresponding training result; subsequently, the loss function can be calculated based on the training result, and the initial recognition model can be corrected according to the loss function to finally obtain the voice recognition model.

[0096] In summary, the speech recognition method proposed in the above steps 101 to 102, on the one hand, uses the speech data after data enhancement processing for model training, which can effectively increase the robustness of the model; on the other hand, in the process of speech recognition, the voiceprint information corresponding to the speech to be recognized is combined to supplement the information of the speech recognition model, thereby improving the accuracy of speech recognition; on the other hand, the speech recognition model including the self-attention network can make full use of the contextual information in the speech, further improving the accuracy of speech recognition.

[0097] That is to say, the speech recognition method proposed in the embodiment of the present application applies the self-attention network structure to the SAE technology. The self-attention network has a strong ability to model the temporal information of speech, so that the model can better learn the contextual information in the speech sequence, thereby improving the model's estimation accuracy of the speaker's age in the user video.

[0098] Furthermore, the speech recognition method proposed in the embodiment of the present application integrates the voiceprint feature information into the speech recognition model, so that the speech recognition model can simultaneously learn the voiceprint features in the speech, which can improve the accuracy of age estimation.

[0099] The embodiment of the present application provides a speech recognition method, wherein the terminal device extracts voiceprint features of the speech to be recognized, determines the first voiceprint information corresponding to the speech to be recognized; inputs the speech to be recognized and the first voiceprint information into the speech recognition model, and obtains the recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training the self-attention network with the enhanced speech data. That is to say, in the embodiment of the present application, the terminal device integrates the voiceprint features of the speech in the process of speech recognition, and improves the accuracy of speech recognition. At the same time, the speech recognition model used by the terminal device for speech recognition processing is trained with the enhanced speech data, which can effectively improve the robustness of the speech recognition model, so that the speech recognition model can be better applied to complex audio scenes, and the speech recognition model can make full use of the contextual information of the speech through the self-attention network, further improving the accuracy of speech recognition. It can be seen that in the embodiment of the present application, based on the combined use of voiceprint features and speech recognition models, the age information of the speaker can be accurately identified, which greatly improves the accuracy of speech recognition, thereby improving the intelligence of the terminal device.

[0100] Based on the above embodiments, the embodiments of the present application provide a speech recognition method, which can improve and optimize the accuracy of speech recognition from three aspects: data enhancement, fusion of voiceprint features, and the use of self-attention network (SAN). Among them, in the training process of the SAE model (speech recognition model), the training data (speech data in the speech data set) is subjected to data enhancement processing, which may specifically include speed change without pitch change, noise addition, reverberation, etc., so as to improve the robustness of the model for complex scenes; in the training process of the SAE model, the speech voiceprint feature vector is added, so that the SAE model can fuse the voiceprint features of the speech to improve the accuracy of speech recognition; a SAE model based on the self-attention network is built, so that the model can make full use of the contextual information of the speech, so as to achieve better speech recognition effect.

[0101] It is understandable that for common scenarios where the speaker's age needs to be accurately identified, for example, using videos in the album to create user video memories based on children's clips, the age is estimated for the audio clips obtained from the video, and the video clips judged to be children's voices will be used as materials for creating children's memories. Therefore, the accuracy of speech recognition has a direct impact on the effect of the final wonderful memories.

[0102] Exemplarily, in some embodiments, Fig.10 This is a schematic diagram of the application scenario of the speech recognition solution, such as Fig.10 As shown in the figure, in the child voice detection module of the album multimodal intelligent editing and search engine project, wonderful memories of the user's album will be created based on children's video clips. Among them, the overall technical framework includes the following parts: 1. Acquire voice data, that is, obtain the album video and extract the audio in the video; 2. Perform active voice monitoring on the voice to be recognized, that is, perform active voice detection on the audio and remove silent and noise fragments; 3. Acquire the voice to be recognized, that is, use speaker segmentation and clustering technology on the voice fragments, segment the voice fragments, and then cluster them according to the voiceprint information of the voice fragments, so as to separate the voices of each speaker; 4. Perform age recognition processing on the voice to be recognized, that is, use SAE to determine whether the voice of each separated speaker belongs to the child's voice; 5. Generate child voice memories, that is, create wonderful memories of video children's voices for the detected children's voice fragments.

[0103] It can be seen that SAE technology is directly responsible for the video material of the wonderful memories of children's voices in the above application scenarios. Therefore, the accuracy and effect of speech recognition play a pivotal role in creating the effect of wonderful memories of children's voices.

[0104] In order to improve the accuracy of speech recognition, the present application embodiment proposes a speech recognition method, which is a speaker age estimation technology integrating voiceprint features. Fig.11 This is a framework diagram for implementing the speech recognition method, such as Fig.11 As shown, it can be divided into three modules: data enhancement, voiceprint recognition system (voiceprint recognition network), and SAE model structure (speech recognition model).

[0105] For the data enhancement module, in order to make the SAE model (speech recognition model) highly robust, in addition to applying the prepared training data, the training data will also be enhanced, including the following methods: changing speed without changing pitch, adding noise, and adding reverberation.

[0106] Since there are a large number of speech with different speaking speeds in the video data, we can choose to perform speed-changing and pitch-invariant processing on the training speech to enable the model to cope with the speed changes of different speech in actual scenarios. The specific speed-changing and pitch-invariant method is: for each speech, speed-changing to 0.9, 1.0, and 1.1 times the original speed, while ensuring that the pitch remains unchanged. Speed-changing and pitch-invariant processing can simulate the changes in the speaker's speaking speed in actual scenarios, thereby improving the robustness of the model.

[0107] The original input speech signal is processed by adding noise using the open source MUSAN dataset, which contains three types of noise: speech, music, and noise. The specific noise adding method is: for each training speech, one type of noise is randomly selected for noise adding. For speech noise, the signal-to-noise ratio of the noise adding is a random value of 0dB-15dB; for music noise, the signal-to-noise ratio of the noise adding is a random value of 5dB-15dB; for noise noise, the signal-to-noise ratio of the noise adding is a random value of 10dB-20dB. This dataset covers many types of noise, and the noise adding process covers different signal-to-noise ratios. Therefore, it can simulate the complex scenes in the user's album videos to a certain extent and improve the accuracy of the model.

[0108] The simulated_rirs reverberation data in the open-source RIRS dataset is used to add reverberation to the speech signal. The specific noise addition method is: randomly select a reverberation data and add reverberation to the input speech. This dataset covers a lot of reverberation data. After adding reverberation, it can better simulate the reverberation scene of real users and improve the accuracy of the model.

[0109] The voiceprint recognition system can integrate the voiceprint information of the speech into the SAE system framework, so that the model can estimate the age attribute of the speech more accurately. Fig.11 As shown in the figure, for each voice, the voiceprint recognition system is used to extract the r-vector vector that can represent the voiceprint features, and then it is fused into the SAE model.

[0110] like Figure 2 As shown in the figure, the network structure of the voiceprint recognition system based on neural network is divided into a frame-level deep feature learning layer, a feature aggregation layer, and a segment-level representation learning layer. The model is trained by speaker-discriminative training criteria, and the speaker representation (Speaker embedding) vector is extracted at the sentence level. Among them, this application adopts a network structure based on ResNet, so the extracted speaker representation vector is referred to as r-vector.

[0111] like Figure 3 As shown in the figure, a 34-layer ResNet network structure is used, the input speech is 2 seconds long, the frame length is set to 25 milliseconds, the frame shift is 10 milliseconds, and the 40-dimensional FABNK (Filter Bank) feature is extracted. Then the dimension of the speech FBANK feature is 40x200x1. Specifically, the first part is a layer of Conv2d, which ensures that the time dimension and frequency dimension remain unchanged and increases the number of channels of the input feature. The second part is a stack of 32 layers of Conv2d (ResNetBlock-1, ResNetBlock-2, ResNetBlock-3, ResNetBlock-4), which convolve the time dimension and frequency dimension to extract the high-dimensional features of the speech. The third part is the StatsPooling layer and the Flatten layer, which are used to extract the mean and variance of the sentence level, so that the feature representation of the frame level is converted to the feature representation of the sentence level. The fourth part is the fully connected neural network Dense1 and Dense2, where Dense1 reduces the feature representation of the sentence level to 256 dimensions, which is the vector representation r-vector of the voiceprint feature. Dense2 maps the r-vector to the final number of speakers N and uses the softmax function to obtain the posterior probability representation of speaker classification.

[0112] like Fig.11 As shown, since the SAE model needs to integrate the voiceprint features of speech, this application divides the SAE network structure into two modules: Self-Attention1 and Self-Attention2, where Self-Attention1 extracts the age attribute information in the speech (called a-vector), and then adds it to the r-vector vector representing the voiceprint, and then uses Self-Attention2 to learn the features of the fusion of a-vector and r-vector.

[0113] Exemplarily, in some embodiments, Fig.12 This is a schematic diagram of the structure of the speech recognition model proposed in the embodiment of the present application, such as Fig.12 As shown in the figure, in the SAE network structure based on Self-Attention, the Self-Attention1 and Slef-Attention2 networks are exactly the same, both of which are composed of N layers of Self-Attention layers stacked (N is an integer greater than 0). Each self-attention layer has two sub-layers. The first is the Multi-Head Attention mechanism (MHA), and the second is a simple Feed Forward Network (FFN). We use Residual Connection (RC) for each sub-layer, and then perform layer normalization (LN). In order to keep the dimension consistent with the voiceprint vector r-vector, all sub-layers in the model produce outputs with a dimension of 256.

[0114] Multi-Head Attention can be described as mapping a query (Query, Q) and a set of key-value pairs (Key-Value) to the output, where the query (Q), key (K), value (V) and output are all vectors. The output is calculated by the weighted sum of the values, where the weight assigned to each value is calculated by the value of Q and the corresponding K. The core idea of ​​MHA is to project Q, K, and V through h different linear transformations, and finally splice the different attention results together. Among them, the scaled dot-product attention mechanism (SDPA) is implemented as follows: a set of queries are packaged into the matrix Q, and the keys and values ​​are also packaged into the matrices K and V respectively.

[0115] The FFN layer in the Self-Attention layer is essentially a two-layer fully connected neural network. The activation function of the first layer is generally a linear rectified unit (ReLU), and the second layer is a linear activation function. If the output of the MHA layer is represented as Z, then FFN can be expressed by the above formula (2).

[0116] Based on the speech recognition method proposed in this application, for the input FBANK features, on the one hand, it will pass through the Self-Attention1 network to extract the a-vector vector (for example, the dimension is 256) representing the age information, and on the other hand, it will pass through the voiceprint recognition network to extract the r-vector vector (for example, the dimension is 256) representing the voiceprint information, and then add the a-vector and r-vector to achieve the purpose of fusing the voiceprint features, and then pass the added vector through the Self-Attention2 network, and finally use FC to map to the output node (recognition result, such as age categories including children, middle-aged, and elderly).

[0117] It can be seen that compared with the SAE method based on traditional machine learning (non-deep learning) and the current mainstream deep learning-based method, in the complex scenarios of user album videos, the speech recognition method proposed in this application can overcome the complexity of background speech diversity and low signal-to-noise ratio scenarios, while making full use of the voiceprint information in the speech. In addition, the SAN network layer is used in the model structure to model the speech timing information.

[0118] On the one hand, in this application, at the front end of the solution, data enhancement processing is performed, mainly using methods such as changing speed without changing pitch, adding noise, and adding reverberation to increase the robustness of the model.

[0119] On the other hand, in this application, at the architectural level, the voiceprint feature vector r-vector of speech is innovatively integrated on the basis of the a-vector age feature vector. For user videos in real scenes, the voiceprint features can supplement the information of the SAE model, thereby improving the accuracy of the model's age estimation. Specifically, the voiceprint model of the ResNet network structure is first trained using voiceprint data, and then the voiceprint model is used to extract the r-vector vector from the speech to represent the voiceprint information, and finally the r-vector and the a-vector vector in the SAE model framework of this scheme are added, so that the model can learn the voiceprint features in the speech at the same time. In practical applications, the SAE model that integrates voiceprint features can improve the accuracy of age estimation.

[0120] On the other hand, in this application, at the model level, the self-attention network structure is innovatively applied to the SAE technology, so that the network structure can better model the speech timing information and make full use of the contextual information in the speech, thereby improving the recognition accuracy of the model. This is because the self-attention network structure is applied to the SAE technology, and the self-attention network has a strong ability to model the timing information of the speech, so that the model can better learn the contextual information in the speech sequence, thereby improving the model's estimation accuracy of the speaker's age in the user's video.

[0121] It should be noted that in the embodiments of the present application, the self-attention network structure is applied to the SAE technology so that the model can better learn the context information in the speech sequence. It is also possible to consider using a gated recurrent unit (GRU) to model the time series speech signal.

[0122] It should be noted that, in the embodiments of the present application, the a-vector and the r-vector may be fused by an addition method, which is relatively simple. It is also possible to consider fusing the two vectors by a dot product or concatenation method.

[0123] The embodiment of the present application provides a speech recognition method, wherein the terminal device extracts voiceprint features of the speech to be recognized, determines the first voiceprint information corresponding to the speech to be recognized; inputs the speech to be recognized and the first voiceprint information into the speech recognition model, and obtains the recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training the self-attention network with the enhanced speech data. That is to say, in the embodiment of the present application, the terminal device integrates the voiceprint features of the speech in the process of speech recognition, and improves the accuracy of speech recognition. At the same time, the speech recognition model used by the terminal device for speech recognition processing is trained with the enhanced speech data, which can effectively improve the robustness of the speech recognition model, so that the speech recognition model can be better applied to complex audio scenes, and the speech recognition model can make full use of the contextual information of the speech through the self-attention network, further improving the accuracy of speech recognition. It can be seen that in the embodiment of the present application, based on the combined use of voiceprint features and speech recognition models, the age information of the speaker can be accurately identified, which greatly improves the accuracy of speech recognition, thereby improving the intelligence of the terminal device.

[0124] Based on the above embodiment, in another embodiment of the present application, Fig.13 Schematic diagram of the structure of the terminal device proposed in the embodiment of the present application Figure 1 ,like Fig.13 As shown, the terminal device 10 proposed in the embodiment of the present application may include a determining unit 111, an acquiring unit 112,

[0125] The determining unit 111 is used to extract voiceprint features of the speech to be recognized and determine first voiceprint information corresponding to the speech to be recognized;

[0126] The acquisition unit 112 is used to input the speech to be recognized and the first voiceprint information into a speech recognition model to obtain a recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training a self-attention network with enhanced speech data.

[0127] In the embodiments of the present application, further, Fig.14 Schematic diagram of the structure of the terminal device proposed in the embodiment of the present application Figure 2 ,like Fig.14 As shown, the terminal device 10 proposed in the embodiment of the present application may also include a processor 121 and a memory 122 storing executable instructions of the processor 121. Further, the terminal device 10 may also include a communication interface 123 and a bus 124 for connecting the processor 121, the memory 122 and the communication interface 123.

[0128] In the embodiment of the present application, the processor 121 can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the function of the processor can also be other, and the embodiment of the present application is not specifically limited. The terminal device 10 can also include a memory 122, which can be connected to the processor 121, wherein the memory 122 is used to store executable program code, the program code includes computer operation instructions, and the memory 122 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0129] In the embodiment of the present application, the bus 124 is used to connect the communication interface 123, the processor 121 and the memory 122, and the mutual communication between these devices.

[0130] In the embodiment of the present application, the memory 122 is used to store instructions and data.

[0131] Furthermore, in an embodiment of the present application, the above-mentioned processor 121 is used to extract voiceprint features of the speech to be recognized, determine the first voiceprint information corresponding to the speech to be recognized; input the speech to be recognized and the first voiceprint information into the speech recognition model to obtain the recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training the self-attention network with enhanced speech data.

[0132] In practical applications, the memory 122 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 121.

[0133] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0134] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., which can store program code.

[0135] The embodiment of the present application provides a terminal device, which extracts voiceprint features of a speech to be recognized, determines the first voiceprint information corresponding to the speech to be recognized, inputs the speech to be recognized and the first voiceprint information into a speech recognition model, and obtains a recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training the self-attention network with enhanced speech data. That is to say, in the embodiment of the present application, the terminal device integrates the voiceprint features of the speech in the process of speech recognition, and improves the accuracy of speech recognition. At the same time, the speech recognition model used by the terminal device for speech recognition processing is trained with enhanced speech data, which can effectively improve the robustness of the speech recognition model, so that the speech recognition model can be better applied to complex audio scenes, and the speech recognition model can make full use of the contextual information of the speech through the self-attention network, further improving the accuracy of speech recognition. It can be seen that in the embodiment of the present application, based on the combined use of voiceprint features and speech recognition models, the age information of the speaker can be accurately identified, which greatly improves the accuracy of speech recognition, thereby improving the intelligence of the terminal device.

[0136] An embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the speech recognition method as described above is implemented.

[0137] Specifically, the program instructions corresponding to a speech recognition method in this embodiment can be stored on a storage medium such as a CD, a hard disk, a USB flash drive, etc. When the program instructions corresponding to a speech recognition method in the storage medium are read or executed by an electronic device, the following steps are included:

[0138] Extracting voiceprint features of the speech to be recognized, and determining first voiceprint information corresponding to the speech to be recognized;

[0139] The speech to be recognized and the first voiceprint information are input into a speech recognition model to obtain a recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training a self-attention network with enhanced speech data.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0141] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0144] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A speech recognition method, characterized in that: The method comprises: Extracting voiceprint features of the speech to be recognized, and determining first voiceprint information corresponding to the speech to be recognized; The speech to be recognized and the first voiceprint information are input into a speech recognition model to obtain a recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training a self-attention network with enhanced speech data.

2. The method according to claim 1, characterized in that The speech recognition model includes a first self-attention network and a second self-attention network, and the inputting the speech to be recognized and the first voiceprint information into the speech recognition model to obtain a recognition result corresponding to the speech to be recognized includes: Extracting attribute features of the speech to be recognized through the first self-attention network to obtain first attribute information corresponding to the speech to be recognized; Determining fused feature information corresponding to the to-be-recognized speech according to the first attribute information and the first voiceprint information; The recognition result corresponding to the speech to be recognized is determined through the second self-attention network and the fused feature information.

3. The method according to claim 2, characterized in that The speech recognition model includes a fully connected layer, and determining the recognition result corresponding to the speech to be recognized through the second self-attention network and the fused feature information includes: Learning the fused feature information through the second self-attention network to obtain feature information corresponding to the speech to be recognized; The feature information corresponding to the speech to be recognized is classified through the fully connected layer to determine the recognition result corresponding to the speech to be recognized.

4. The method according to claim 2 or 3, characterized in that: The first self-attention network and the second self-attention network both include multiple layers of self-attention layers; wherein the self-attention layers include a multi-head self-attention mechanism network and a feedforward network.

5. The method according to claim 3, characterized in that: The determining, according to the first attribute information and the first voiceprint information, fused feature information corresponding to the speech to be recognized includes: The first attribute information and the first voiceprint information are fused according to a preset fusion strategy to obtain the fused feature information; wherein the preset fusion strategy includes additive fusion, or dot product fusion, or concatenation fusion.

6. The method according to claim 5, characterized in that The method further comprises: Performing data enhancement processing on the speech data in the speech data set to obtain the enhanced speech data; The enhanced speech data is used to train an initial recognition model to obtain the speech recognition model.

7. The method according to claim 6, characterized in that The data enhancement processing at least includes speed change processing, noise addition processing, and reverberation processing; in, The speed change processing is used to change the speech speed of the voice data according to a preset speech speed parameter; The noise adding process is used to add noise to the speech data according to the noise data; The reverberation processing is used to add reverberation to the speech data according to the reverberation data.

8. The method according to claim 6, characterized in that The initial recognition model includes the first self-attention network, the second self-attention network and the fully connected layer, and the initial recognition model is trained using the enhanced speech data to obtain the speech recognition model, including: Using a voiceprint recognition network to perform voiceprint recognition processing on the voice data in the voice data set, and determining second voiceprint information corresponding to the enhanced voice data; Extracting attribute features of the enhanced speech data through the first self-attention network to obtain second attribute information corresponding to the enhanced speech data; Learning the fusion feature through the second self-attention network to determine the feature information corresponding to the enhanced voice data; wherein the fusion feature is obtained by fusing the second attribute information and the second voiceprint information; Classifying the feature information corresponding to the enhanced speech data through the fully connected layer to determine the training result corresponding to the enhanced speech data; A loss function is calculated based on the training result, and the initial recognition model is corrected according to the loss function to obtain the speech recognition model.

9. The method according to claim 8, characterized in that The step of extracting voiceprint features from the speech to be recognized and determining first voiceprint information corresponding to the speech to be recognized includes: The voiceprint recognition network is used to perform voiceprint recognition processing on the speech to be recognized to determine the first voiceprint information.

10. The method according to claim 8 or 9, characterized in that: The voiceprint recognition network is a ResNet network structure, and the voiceprint recognition network includes a frame-level feature learning layer, a feature aggregation layer, and a sentence-level feature learning layer.

11. A terminal device, characterized in that: The terminal device comprises: a determination unit, an acquisition unit, The determining unit is used to extract voiceprint features of the speech to be recognized and determine first voiceprint information corresponding to the speech to be recognized; The acquisition unit is used to input the speech to be recognized and the first voiceprint information into a speech recognition model to obtain a recognition result corresponding to the speech to be recognized; wherein the speech recognition model is obtained by training a self-attention network with enhanced speech data.

12. A terminal device, characterized in that: The terminal device includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images