Speech recognition model training method, speech recognition method and device

Abstract

The present application provides a speech recognition model training method, a speech recognition method and a device, which include: obtaining a first training sample set during any iterative process, wherein each training data in the first training sample set includes speech data, a transcribed text of the speech data and historical speech data of the speech data; the speech recognition model includes a feature extraction model and a recognition model; the speech data in the training data and the historical speech data of the speech data are respectively used as inputs of the pre-trained feature extraction model, and the context feature vector of the speech data and the context feature vector of the historical speech data are outputted; the speech data, the context feature vector of the speech data and the context feature vector of the historical speech data are used as inputs of the recognition model, and the recognition text of the speech data is outputted; and the parameters of the recognition model are adjusted according to the recognition text of each speech data and the transcribed text of each speech data obtained in each iterative process until the training stop condition is met.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a speech recognition model training method, a speech recognition method, and a speech recognition device. Background Art

[0002] Automatic speech recognition (ASR) is the process of converting audio into text. With the advancement of computer technology and artificial intelligence, speech recognition is being applied in an increasing number of scenarios. Within ASR tasks, end-to-end speech recognition is a current research hotspot.

[0003] Current speech recognition technology has low recognition accuracy for long-tail entities (i.e., less frequently occurring words or professional terms in specific fields) in long speech recognition scenarios such as intelligent customer service systems, smartphone assistants, automatic video subtitle generation, automatic speech-to-text conversion in instant messaging software, and online voice interaction. Summary of the Invention

[0004] The present application provides a speech recognition model training method, a speech recognition method and a device, which can improve the accuracy of speech recognition, especially improve the recognition accuracy of long-tail entities in long speech recognition scenarios.

[0005] In a first aspect, the present application provides a speech recognition model training method, comprising:

[0006] During any iteration of speech recognition model training, a first training sample set is obtained, where the first training sample set includes a plurality of training data, each training data includes speech data, a transcribed text of the speech data, and historical speech data of the speech data, and the speech recognition model includes a feature extraction model and a recognition model;

[0007] For each training data in the first training sample set, using speech data in the training data as input to the feature extraction model, outputting a context feature vector of the speech data, and using historical speech data of the speech data as input to the feature extraction model, outputting a context feature vector of the historical speech data, wherein the feature extraction model is obtained through pre-training;

[0008] Taking the speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data as inputs of the recognition model, and outputting a recognition text of the speech data;

[0009] According to the recognition text of each speech data in the first training sample set and the transcribed text of each speech data obtained in each iterative process, the parameters of the recognition model are adjusted until the training stop condition is met, and the speech recognition model determined by the iterative process that meets the training stop condition is determined as the trained speech recognition model.

[0010] In a second aspect, the present application provides a speech recognition method, comprising:

[0011] Acquire voice signals;

[0012] The speech signal and the historical speech signal of the speech signal are input into a pre-trained speech recognition model, and a speech recognition result of the speech signal is output. The speech recognition model is trained according to the method described in the first aspect.

[0013] In a third aspect, the present application provides a speech recognition model training device, comprising:

[0014] an acquisition module, configured to acquire a first training sample set during any iteration of speech recognition model training, wherein the first training sample set includes a plurality of training data, each training data including speech data, a transcribed text of the speech data, and historical speech data of the speech data, and the speech recognition model includes a feature extraction model and a recognition model;

[0015] a first processing module, configured to, for each training data in the first training sample set, use speech data in the training data as input to the feature extraction model and output a context feature vector of the speech data, and use historical speech data of the speech data as input to the feature extraction model and output a context feature vector of the historical speech data, wherein the feature extraction model is obtained through pre-training;

[0016] a second processing module, configured to use the speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data as inputs of the recognition model, and output a recognized text of the speech data;

[0017] A parameter adjustment module is used to adjust the parameters of the recognition model according to the recognition text of each speech data in the first training sample set and the transcription text of each speech data obtained in each iterative process until the training stop condition is met, and the speech recognition model determined by the iterative process that meets the training stop condition is determined as the trained speech recognition model.

[0018] In a fourth aspect, the present application provides a speech recognition device, comprising:

[0019] An acquisition module, used for acquiring a voice signal;

[0020] The speech recognition module is used to input the speech signal and the historical speech signal of the speech signal into a pre-trained speech recognition model, and output a speech recognition result of the speech signal. The speech recognition model is trained according to the method described in the first aspect.

[0021] In a fifth aspect, the present application provides a computer device comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method of the first aspect or the second aspect.

[0022] In a sixth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer program, enable the computer to execute the method of the first aspect or the second aspect.

[0023] In a seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the method of the first aspect or the second aspect.

[0024] In summary, in the present application, a speech recognition model including a feature extraction model and a recognition model is trained. In any iterative process, the first training sample set used for model training includes multiple training data, each training data includes speech data, a transcribed text of the speech data, and historical speech data of the speech data. The feature extraction model is obtained through pre-training. The feature extraction model directly extracts the context feature vector of the historical speech data from the historical speech data of the speech data, and directly extracts the context feature vector of the speech data from the speech data. The speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data of the speech data are then used as inputs of the recognition model to output the recognition text of the speech data. Then, based on the recognition text of each speech data and the transcribed text of each speech data in the first training sample set obtained in each iterative process, the parameters of the recognition model are adjusted until the training stop condition is met, and the speech recognition model determined by the iterative process that meets the training stop condition is determined as the trained speech recognition model. Since the pre-trained feature extraction model can directly extract the context feature vector of speech data from speech data, and extract the context feature vector of historical speech data from the historical speech data of speech data, it can avoid the problem of inputting the recognition results of historical speech into the language model to extract context features in the existing technology, which has the problem of accumulated recognition errors leading to low speech recognition accuracy. Moreover, the extracted context feature vector is a high-level information representation related to the context. The speech data, the context feature vector of the speech data and the context feature vector of the historical speech data are jointly used as the input of the recognition model, which enables the recognition model to effectively utilize the context information during the learning process. Therefore, the trained speech recognition model can improve the accuracy of speech recognition, especially improve the recognition accuracy of long-tail entities in long speech recognition scenarios.

[0025] Furthermore, in the present application, the feature extraction model includes a speech encoder, a text encoder, and a cross-modal encoder, wherein at least one of the speech encoder and the text encoder can be a pre-trained model. By adopting the pre-trained model, the model parameters of the speech encoder and the text encoder can be trained by a large amount of unlabeled training data, and the training data can come from various fields. Therefore, the pre-trained model has good robustness and generalization, which in turn can make the speech recognition model have good robustness and generalization. By adopting the pre-trained model, the feature extraction model only needs a small amount of training data to obtain an effective context feature vector, so that the speech recognition method of the embodiment of the present application can be applied to low-resource speech recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A diagram illustrating a speech recognition model training method and an implementation scenario of the speech recognition method provided in an embodiment of the present application;

[0027] Figure 2 A flowchart of a speech recognition model training method provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the structure of a speech recognition model provided in an embodiment of the present application;

[0029] Figure 4 A flowchart of a feature extraction model training method provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the structure of a feature extraction model provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of obtaining a second context feature vector of training data provided in an embodiment of the present application;

[0032] Figure 7 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of the structure of a speech recognition model provided in an embodiment of the present application;

[0034] Figure 9 A schematic diagram of the structure of a speech recognition model training device provided in an embodiment of the present application;

[0035] Figure 10 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0036] Figure 11 It is a schematic block diagram of a computer device 700 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0039] Before introducing the technical solution of this application, the following is an introduction to the relevant knowledge of this application:

[0040] 1. Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0041] 2. Machine Learning (ML): This is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0042] 3. Deep Learning (DL): A branch of machine learning, it is an algorithm that attempts to achieve high-level abstraction of data using multiple processing layers containing complex structures or consisting of multiple nonlinear transformations. Deep learning learns the inherent patterns and representational hierarchies of training sample data. The information gained during this learning process is highly helpful in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to have human-like analytical learning capabilities and to recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far surpass previous related technologies.

[0043] 4. Pretraining: The process of training a neural network model using a large dataset to enable it to learn the common features of the dataset. The purpose of pretraining is to provide high-quality model parameters for subsequent training of the neural network model on a specific dataset. In the embodiments of this application, at least one of the speech encoder and text encoder can be a pretrained model.

[0044] The speech recognition model training method provided in the embodiments of this application mainly relates to speech technology in artificial intelligence technology, specifically ASR technology. The following embodiments can be used to illustrate the details. The key technologies of speech technology include ASR technology, text-to-speech technology (TTS), and voiceprint recognition technology. Enabling computers to see, hear, speak, and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0045] In the existing technology, context features are usually used to improve the accuracy of speech recognition. The existing context feature extraction method uses the language model in the speech recognition system to extract context features. During the speech recognition model training process, context features are extracted from the manually transcribed text corresponding to the historical speech (the correct transcribed text). However, in actual use, the speech recognition model cannot obtain the correct transcribed text of the historical speech. Usually, the recognition results of the historical speech are input into the language model to extract context features. The recognition results of the historical speech usually contain recognition errors, which will cause noise in the context features extracted by the language model, thereby introducing deviations in the recognition process of the current speech, resulting in low speech recognition accuracy. In order to solve this technical problem, the embodiment of the present application trains a speech recognition model including a feature extraction model and a recognition model. In any iterative process of model training, the first training sample set used for model training includes multiple training data, each training data includes speech data, transcribed text of speech data and historical speech data of speech data. The feature extraction model is obtained through pre-training. The feature extraction model directly extracts the context feature vector of historical speech data from the historical speech data of speech data, and directly extracts the context feature vector of speech data from speech data. The speech data, the context feature vector of speech data and the context feature vector of historical speech data of speech data are then used as inputs of the recognition model to output the recognition text of speech data. Then, according to the recognition text of each speech data and the transcribed text of each speech data in the first training sample set obtained in each iterative process, the parameters of the recognition model are adjusted until the training stop condition is met, thereby obtaining a trained speech recognition model. Since the pre-trained feature extraction model can directly extract the context feature vector of speech data from speech data, and extract the context feature vector of historical speech data from the historical speech data of speech data, it can avoid the problem of inputting the recognition results of historical speech into the language model to extract context features in the existing technology, which has the problem of accumulated recognition errors leading to low speech recognition accuracy. Moreover, the extracted context feature vector is a high-level information representation related to the context. The speech data, the context feature vector of the speech data and the context feature vector of the historical speech data are jointly used as the input of the recognition model, which enables the recognition model to effectively utilize the context information during the learning process. Therefore, the trained speech recognition model can improve the accuracy of speech recognition, especially improve the recognition accuracy of long-tail entities in long speech recognition scenarios.

[0046] Furthermore, the feature extraction model in the embodiment of the present application includes a speech encoder, a text encoder, and a cross-modal encoder, wherein at least one of the speech encoder and the text encoder can be a pre-trained model. By adopting the pre-trained model, the model parameters of the speech encoder and the text encoder can be trained by a large amount of unlabeled training data, and the training data can come from various fields. Therefore, the pre-trained model has good robustness and generalization, which in turn can make the speech recognition model have good robustness and generalization. By adopting the pre-trained model, the feature extraction model only needs a small amount of training data to obtain an effective context feature vector, so that the speech recognition method of the embodiment of the present application can be applied to low-resource speech recognition scenarios.

[0047] The speech recognition model training method and speech recognition method provided in the embodiments of this application can be applied to various long speech recognition scenarios, such as intelligent customer service speech recognition systems, smartphone assistants, automatic video subtitle generation, automatic speech-to-text conversion in instant messaging software, and online voice interaction. They can significantly improve the recognition accuracy of long-tail entities in long speech recognition scenarios, enhancing the user experience. They can also be applied to other speech recognition scenarios, which are not limited by this application.

[0048] For example, Figure 1 A diagram of an application scenario of a speech recognition model training method and a speech recognition method provided in an embodiment of the present application, such as Figure 1 As shown, the implementation scenario of the embodiment of the present application involves a server 1 and a terminal device 2, and the terminal device 2 can communicate data with the server 1 through a communication network.

[0049] In some implementations, terminal device 2 refers to a device that has a rich set of human-computer interaction methods, has internet access, is typically equipped with various operating systems, and has strong processing capabilities. The terminal device may be, but is not limited to, a smartphone, tablet computer, portable laptop computer, desktop computer, portable wearable device, smart speaker, or vehicle-mounted terminal. Optionally, in an embodiment of the present application, a voice recognition software client is installed in terminal device 2, and the user can use this client to input the corresponding voice information to be recognized.

[0050] In some implementations, the terminal device 2 includes, but is not limited to, a smartphone, a tablet computer, an intelligent voice interaction device, a smart home appliance, an in-vehicle terminal, etc. For example, the intelligent voice interaction device may be a smart speaker, a smart TV box, an online voice interaction system, an intelligent voice assistant, an in-vehicle intelligent voice device, an intelligent voice device with simultaneous interpretation function or equipped with a voice input method, etc.

[0051] Figure 1Server 1 in the description may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery services (Content Delivery Network, CDN), and big data and artificial intelligence platforms. This application does not impose any restrictions on this.

[0052] Exemplarily, in some possible implementations, server 1 is used to deploy a speech recognition model and train the speech recognition model, and deploy the trained speech recognition model in a corresponding terminal device, and use the deployed speech recognition model through the terminal device (such as terminal device 2) to process voice information in the usage environment, such as performing speech recognition.

[0053] It is understood that before the speech recognition model processes the speech information in the use environment, the speech recognition model needs to be trained. Specifically, the speech recognition model training method provided in the embodiment of the present application can be used. The speech recognition model training method provided in the embodiment of the present application can improve the accuracy of speech recognition, especially the recognition accuracy of long-tail entities in long speech recognition scenarios.

[0054] In some implementations, Figure 1 One terminal device and one server are shown as an example, but other numbers of terminal devices and servers may actually be included, and this application does not impose any limitation on this.

[0055] The speech recognition model training method provided in the embodiment of the present application may be executed by the speech recognition model training device provided in the embodiment of the present application, or a computer device integrated with the speech recognition model training device, wherein the speech recognition model training device may be implemented in hardware or software. The computer device may be Figure 1 The terminal device 2 or server 1 in.

[0056] In some possible implementations, the speech recognition model training method provided in the embodiments of the present application can use servers, workstations and other devices that include computing hardware such as CPU, GPU or TPU to train the speech recognition model, or use a server cluster or distributed system composed of multiple physical servers to train the speech recognition model. It can also use a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms to train the speech recognition model.

[0057] The technical solutions provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0058] Figure 2 A flowchart of a speech recognition model training method provided in an embodiment of the present application, the execution subject of the method may be a speech recognition model training device, such as Figure 2 As shown, the method may include:

[0059] S101. During any iteration of speech recognition model training, a first training sample set is obtained. The first training sample set includes multiple training data. Each training data includes speech data, transcribed text of speech data, and historical speech data of speech data. The speech recognition model includes a feature extraction model and a recognition model.

[0060] Specifically, in this embodiment, when training the speech recognition model, the model parameters need to undergo multiple iterations until the training stop condition is met, for example, until the model converges. In any iterative process of speech recognition model training, the training process of S101-S104 is executed until the training stop condition is met, thereby obtaining a trained speech recognition model. The first training sample set is the training sample used for one training session. The first training sample set can be obtained by selecting a preset number of training samples from the first original training sample set to form the first training sample set. The preset number can be set according to the actual training, and this application does not impose any restrictions on this.

[0061] The first training sample set includes multiple training data, each of which includes three parts of data, namely, speech data, transcription text of speech data and historical speech data of speech data. For example, the first training sample set D1 = {(S1, Y1, S0), (S2, Y2, S1), ..., (S T ,Y t , S T-1 )}, including T training data "audio-text-audio", the tth training data includes voice data S t 、S t Transcript of Y t and S t Historical speech data S t-1 The historical speech data of the speech data is the speech data of the previous sentence of the speech data. Optionally, the speech data and its historical speech data may be two speech signals that are continuous in time.

[0062] Among them, the voice data S t is the speech feature sequence corresponding to the speech signal, that is Indicates voice data S tThe nth frame speech feature (a frame speech feature can be 10ms). The embodiment of the present application does not limit the specific form of the speech feature. For example, effective speech features such as Fbank, Mel Frequency Cepstral Coefficents (MFCC), PLP, PNCC, PCEN, etc. can be used.

[0063] S102. For each training data in the first training sample set, the speech data in the training data is used as the input of the feature extraction model, and the context feature vector of the speech data is output. The historical speech data of the speech data is used as the input of the feature extraction model, and the context feature vector of the historical speech data is output. The feature extraction model is obtained through pre-training.

[0064] S103 , using the speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data as inputs of the recognition model, and outputting the recognition text of the speech data.

[0065] Specifically, the recognition model predicts the recognition text of the speech data based on the input speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data.

[0066] Specifically, in one practicable manner, the recognition model includes an encoder and a decoder. In S103, the recognition model uses the speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data as inputs, and outputs the recognized text of the speech data, which may specifically include:

[0067] S1031 . Use the speech data as input to the encoder, and output a speech feature vector of the speech data.

[0068] S1032. Take the speech feature vector of the speech data, the context feature vector of the speech data and the context feature vector of the historical speech data as the input of the decoder, and output the recognition text of the speech data. The decoder is used to predict the recognition text of the speech data based on the speech feature vector of the speech data, the context feature vector of the speech data and the context feature vector of the historical speech data through the attention mechanism.

[0069] For example, Figure 3 A schematic diagram of the structure of a speech recognition model provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the speech recognition model includes a feature extraction model and a recognition model. The recognition model includes an encoder and a decoder. For the t-th training data (including speech data S t 、S t Transcript of Y t and S t Historical speech data S t-1), during training, the speech data S in the training data t It is the input of the feature extraction model and outputs the context feature vector H of the speech data. t , H t It can be expressed by the following formula:

[0070] H t =Encoder c (A t )=Encoder c (Encoder s (S t ))

[0071] Among them, Encoder c It is a cross-modal encoder, Encoder s It is a speech coder.

[0072] Take the historical speech data S of speech data t-1 It is the input of the feature extraction model and outputs the context feature vector H of the historical speech data. t-1 , H t-1 It can be expressed by the following formula:

[0073] H t-1 =Encoder c (A t-1 )=Encoder c (Encoder s (S t-1 )).

[0074] Then take voice data S t The input of the encoder is the speech feature vector Z of the output speech data t , Z t It can be expressed by the following formula:

[0075] Z t =FFN(Conv(MHSA(S t ))),

[0076] Among them, MHSA stands for multi-head self-attention mechanism, Conv stands for convolution operation, and FFN stands for affine transformation.

[0077] Next, the speech feature vector Z of the speech data is used t , the context feature vector H of the speech data t and the context feature vector H of the historical speech data t-1 It is the input of the decoder and outputs the recognized text of the speech data

[0078] Specifically, the decoder is based on the speech feature vector Z of the speech data. t , the context feature vector H of the speech data t and the context feature vector H of the historical speech data t-1 Predict the speech data S t Recognition text

[0079] Optional, It can be expressed by the following formula:

[0080]

[0081]

[0082] Among them, MHA represents multi-head attention mechanism (multi-head attention or multi-head crossattention), [H t ;H t-1 ] means H t and H t-1 Concatenate in the feature dimension. is the predicted value of the character distribution output by the recognition model at time k, that is, the probability of each character appearing in the character space V where Y is located. Assume that the character space V contains three characters: a, b, and c, for example A possible value of is (0.1, 0.7, 0.2). is the predicted value at the kth moment After the argmax operation, the character with the highest probability is selected, where argmax represents the vector The dimension or subscript corresponding to the maximum value. As the character sequence Y t Character at time k The predicted value is , that is, the character b corresponding to 0.7 is selected as the output of the recognition model at time t, that is, the recognition text output by the recognition model is character b.

[0083] S104. Adjust the parameters of the recognition model according to the recognition text of each speech data and the transcription text of each speech data in the first training sample set obtained in each iterative process until the training stop condition is met, and determine the speech recognition model determined by the iterative process that meets the training stop condition as the trained speech recognition model.

[0084] Specifically, as an implementable approach, adjusting the parameters of the recognition model according to the recognized text of each speech data and the transcribed text of each speech data in the first training sample set may specifically include:

[0085] S1041: Construct a loss function based on the recognized text of each speech data and the transcribed text of each speech data in the first training sample set.

[0086] For example, the loss function may be a cross entropy loss function, and the loss function constructed according to the recognized text of each speech data and the transcribed text of each speech data in the first training sample set may be as shown in the following formula:

[0087]

[0088] Among them, Y t is the voice data S t The transcript of is the voice data S t The recognition text, loss function L ASR is the sum of the cross entropy losses of the recognized text of the speech data and the transcribed text of the speech data in all the training data in the first training sample set. For example, if there are N training data in the first training sample set, the loss function L ASR is the sum of N cross entropy losses.

[0089] S1042. According to the loss function, adjust the parameters of the recognition model through back propagation.

[0090] Specifically, the loss function is the above L ASR For example, the parameters of the recognition model are adjusted by backpropagation according to the sum of N cross entropy losses so that the sum of the cross entropy losses is within a preset range.

[0091] Specifically, after multiple iterations of the above training, the training is stopped until a stopping condition is met. For example, the training is stopped when the loss value of the loss function drops to a first value and the first value no longer changes.

[0092] The speech recognition model training method provided in this embodiment trains a speech recognition model including a feature extraction model and a recognition model. During any iteration of model training, the first training sample set used for model training includes multiple training data, each of which includes speech data, a transcribed text of the speech data, and historical speech data of the speech data. The feature extraction model is obtained through pre-training. The feature extraction model directly extracts context feature vectors of the historical speech data from the historical speech data of the speech data, and directly extracts context feature vectors of the speech data from the speech data. The speech data, the context feature vectors of the speech data, and the context feature vectors of the historical speech data of the speech data are then used as inputs of the recognition model, which outputs a recognized text of the speech data. The parameters of the recognition model are then adjusted based on the recognized text of each speech data and the transcribed text of each speech data in the first training sample set obtained during each iteration until a training stop condition is met. The speech recognition model determined by the iteration that meets the training stop condition is determined as the trained speech recognition model. Since the pre-trained feature extraction model can directly extract the context feature vector of speech data from speech data, and extract the context feature vector of historical speech data from the historical speech data of speech data, it can avoid the problem of inputting the recognition results of historical speech into the language model to extract context features in the existing technology, which has the problem of accumulated recognition errors leading to low speech recognition accuracy. Moreover, the extracted context feature vector is a high-level information representation related to the context. The speech data, the context feature vector of the speech data and the context feature vector of the historical speech data are jointly used as the input of the recognition model, which enables the recognition model to effectively utilize the context information during the learning process. Therefore, the trained speech recognition model can improve the accuracy of speech recognition, especially improve the recognition accuracy of long-tail entities in long speech recognition scenarios.

[0093] In the embodiment of the present application, the feature extraction model is obtained by pre-training. Figure 3 The training process of the feature extraction model is described in detail. Figure 2 On the basis of the method shown in the figure, the method of this embodiment may further include the following steps before performing speech recognition model training: Figure 3 The steps shown are the process of feature extraction model training.

[0094] Figure 4 This is a flow chart of a feature extraction model training method provided in an embodiment of the present application. The execution subject of this method can be a speech recognition model training device, such as Figure 4 As shown, the method may include:

[0095] S201. In any iterative process of feature extraction model training, a second training sample set is obtained, where the second training sample set includes a plurality of training data, and each training data includes speech data and a transcribed text of the speech data.

[0096] Specifically, in this embodiment, when training the feature extraction model, the feature extraction model parameters need to undergo multiple iterations until the training stop condition is met, for example, until the model converges. During any iteration, the training process of S201-S203 is executed until the training stop condition is met, and the feature extraction model determined by the iteration process that meets the training stop condition is determined as the trained feature extraction model. The second training sample set is the training sample used for one training. The second training sample set can be obtained by selecting a preset number of training samples from the second original training sample set to form the second training sample set. The preset number can be set according to the actual training, and this application does not impose any restrictions on this.

[0097] Optionally, the first original training sample set and the second original training sample set may be the same or different.

[0098] During the training process of the feature extraction model, each training data included in the second training sample set includes speech data and the transcription text of the speech data. For example, the second training sample set D2 = {(S1, Y1), (S2, Y2), ..., (S T ,Y T )}, including T training data "audio-text", the tth training data includes voice data S t and S t Transcript of Y t . S t Transcript of Y t , which is the converted text corresponding to the speech signal.

[0099] Among them, the voice data S t is the speech feature sequence corresponding to the speech signal, that is Indicates voice data S t The nth frame speech feature (a frame speech feature can be 10ms). The embodiment of the present application does not limit the specific form of the speech feature, for example, effective speech features such as Fbank, MFCC, PLP, PNCC, PCEN, etc. can be used.

[0100] S202. For each training data in the second training sample set, use the training data as the input of the feature extraction model and output the first context feature vector of the training data; use the training data after modal masking processing as the input of the feature extraction model and output the second context feature vector of the training data.

[0101] Specifically, for each training data in the second training sample set, the first context feature vector and the second context feature vector of the training data are obtained through the feature extraction model. The first context feature vector is the output of the training data input to the feature extraction model, and the second context feature vector is the output of the training data input to the feature extraction model after modal masking processing.

[0102] Optionally, the feature extraction model includes a speech encoder, a text encoder, and a cross-modal encoder.

[0103] In one practicable manner, in S202, using the training data as input to the feature extraction model and outputting the first context feature vector of the training data may specifically include:

[0104] S2021. Use the speech data in the training data as input to the speech encoder, and output a first speech feature vector of the training data.

[0105] S2022. Use the transcribed text of the speech data as input to a text encoder, and output a first text feature vector of the training data.

[0106] S2023. Take the first speech feature vector of the training data and the first text feature vector of the training data as input to a cross-modal encoder, and output a first context feature vector of the training data. The cross-modal encoder is used to associate the first speech feature vector of the training data with the first text feature vector of the training data.

[0107] For example, Figure 5 A structural diagram of a feature extraction model provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the feature extraction model includes a speech encoder, a text encoder and a cross-modal encoder, with the tth training data (including speech data S t and S t Transcript of Y t ) as an example, taking the speech data S in the training data t is the input of the speech encoder and outputs training data S t The first speech feature vector A t , with the transcription text Y of the speech data t It is the input of the text encoder and outputs the first text feature vector E of the training data. t . Take the first speech feature vector A of the training data t and the first text feature vector E of the training data t is the input of the cross-modal encoder and outputs the first context feature vector H of the training data t The cross-modal encoder is used to transform the first speech feature vector A of the training data tand the first text feature vector E of the training data t Associating them so that they can represent each other, that is, the first speech feature vector A t (also called the implicit feature representation of the speech modality) can represent the first text feature vector E t (also called the implicit feature representation of the text modality), the implicit feature representation of the text modality can represent the information of the implicit feature representation of the speech modality, the first context feature vector H t It can be expressed as follows:

[0108] H t =Encoder c (A t ;E t )=Encoder c (Encoder s (S t ); Encoder y (Y t )),

[0109] Among them, Encoder c It is a cross-modal encoder, Encoder s It is a speech encoder. y is a text encoder.

[0110] Optionally, the training data after modality masking processing includes speech data after modality masking processing and transcribed text data after modality masking processing. In S202, before the training data after modality masking processing is used as input to the feature extraction model and the second context feature vector of the training data is output, the method of this embodiment may further include: obtaining the training data after modality masking processing, Figure 6 A schematic diagram of obtaining a second context feature vector of training data provided in an embodiment of the present application, combined with Figure 6 , the following S1-S4 is a specific implementation method for obtaining training data after modal masking processing.

[0111] S1. Perform frame random masking on the speech feature sequence corresponding to the speech data in the training data to obtain the speech feature sequence after frame masking.

[0112] Specifically, with voice data S t For example, S t is the speech feature sequence corresponding to the speech signal, that is Indicates voice data S t The nth frame speech feature (a frame speech feature can be 10ms, for example), the speech feature sequence corresponding to the speech data Perform frame random masking, and each frame speech feature is masked with a preset probability p s Masked, get the speech feature sequence after frame masking The mask of the n-th frame feature is a random variable that follows a Bernoulli distribution, that is For example, a feature sequence of length 5 The corresponding mask sequence is 1, 0, 1, 1, 0, then the masked speech feature sequence is like Figure 6 As shown, the voice data S t The masked speech feature sequence is obtained by random masking of frames

[0113] S2. Determine the product of the masked speech feature sequence and the speech modality mask as the speech data after modality masking processing.

[0114] Specifically, the speech modality masking process refers to each speech data S in the second training data set. T With a preset probability p m Masked, speech modality mask m cs ~Bernoulli(p m ), when the m corresponding to a certain voice data cs = 0, the speech modality is masked, and the input of the speech encoder is an empty feature sequence φ. The input of the feature extraction model is only the text sequence Y t .

[0115] like Figure 6 As shown, the voice data S t The masked speech feature sequence is obtained by random masking of frames Then, after modal random masking, the speech data after modal masking processing is obtained.

[0116] S3. Perform random character masking on the character sequence corresponding to the transcribed text of the speech data to obtain a masked character sequence.

[0117] Specifically, the transcribed text Y of the speech data t It can usually be represented as a sequence of characters. For English, Y t It is usually represented as a valid subword sequence such as BPE (Byte pair encoding) or wordpiece. t It is usually represented as a sequence of Chinese characters. This application does not limit the form of the character sequence. t can be represented as any valid sequence of characters, i.e. Similar to the speech data, the transcribed text Y t First, the characters are randomly masked, and each character is masked with a preset probability p. t Masked, get the character sequence after character masking The mask of the mth character is a random variable that follows a Bernoulli distribution, that is like Figure 6 As shown, the transcription text Y of the speech data t The masked character sequence obtained by random masking of characters

[0118] S4. Determine the product of the masked character sequence and the text modality mask as the transcribed text data after modality masking processing.

[0119] Specifically, the text modality masking process refers to the transcription text Y of each speech data in the second training dataset. T With a preset probability p m Masked, speech modality mask m cy ~Bernoulli(p m ), when the transcription text of a certain speech data corresponds to m cy = 0, the text modality is masked, and the input of the text encoder is an empty feature sequence φ. The input of the feature extraction model is only the speech feature sequence S t .like Figure 6 As shown, the transcription text Y of the speech data t The masked character sequence obtained by random masking of characters After modal random masking, the transcribed text data after modal masking is obtained

[0120] In one practicable manner, in S202, the training data after modality masking is used as input to the feature extraction model, and the second context feature vector of the training data is output. Specifically, the following steps may be performed:

[0121] S2021′: Use the speech data after modal masking processing as input to the speech encoder, and output a second speech feature vector of the training data.

[0122] Specifically, if Figure 6 As shown, the speech data after modal masking (or φ) is the input of the speech encoder, and the second speech feature vector of the output training data is The speech encoder maps the speech signal from the feature space to the latent feature space.

[0123] S2022′: Use the transcribed text data after modal masking processing as input to the text encoder, and output a second text feature vector of the training data.

[0124] Specifically, if Figure 6 As shown, the transcribed text data after modality masking (or φ) is the input of the text encoder, and the second text feature vector of the output training data is The text encoder maps the transcribed text from the character space to the latent feature space.

[0125] S2023', taking the second speech feature vector of the training data and the second text feature vector of the training data as input of the cross-modal encoder, outputting the second context feature vector of the training data, the cross-modal encoder is used to associate the second speech feature vector of the training data with the second text feature vector of the training data.

[0126] Specifically, the cross-modal encoder encodes the second speech feature vector of the training data and the second text feature vector of the training data Associate them so that they can represent each other, that is, the second speech feature vector (also known as the implicit feature representation of the speech modality) can represent the second text feature vector (also called the implicit feature representation of the text modality), the implicit feature representation of the text modality can represent the information of the implicit feature representation of the speech modality, the second context feature vector It can be expressed as follows:

[0127]

[0128] Among them, Encoder c It is a cross-modal encoder, Encoder s It is a speech encoder. y is a text encoder. Figure 6 As shown, the second speech feature vector of the training data and the second text feature vector of the training data It is the input of the cross-modal encoder and outputs the second context feature vector of the training data

[0129] S203. Adjust the parameters of the feature extraction model according to the first context feature vector of each training data and the second context feature vector of each training data in the second training sample set obtained in each iterative process until the training stop condition is met, and determine the feature extraction model determined by the iterative process that meets the training stop condition as the trained feature extraction model.

[0130] In one practicable manner, adjusting the parameters of the feature extraction model according to the first context feature vector of each training data and the second context feature vector of each training data in the second training sample set in S203 may specifically include:

[0131] S2031 : Construct a first loss function according to the first context feature vector of each training data and the second context feature vector of each training data in the second training sample set.

[0132] Optionally, the first loss function L s It can be expressed by the following formula:

[0133]

[0134] Among them, sim represents the cosine similarity function.

[0135]

[0136] H t =Encoder c (A t ;E t )=Encoder c (Encoder s (S t ); Encoder y (Y t ));

[0137] Indicates that the second training sample set contains the training data (S t ,Y t ) (i.e., any training data with j≠t).

[0138] S2032. Construct a second loss function based on the masked characters in the character sequence corresponding to the transcribed text of the speech data in each training data and the masked characters in the character sequence predicted by the feature extraction model, and the masked characters in the predicted character sequence are the output of the second context feature vector of the training data through the output layer.

[0139] Specifically, the second loss function L y It can be expressed by the following formula:

[0140]

[0141] in, Represents the masked characters in the character sequence corresponding to the transcribed text of the speech data in the training data, that is, the mask with a value of 0 The corresponding character can be known based on the masking probability. Represents the masked characters in the character sequence predicted by the feature extraction model. Figure 6 As shown, the masked characters in the predicted character sequence is the second context feature vector of the training data Output through the output layer. Figure 6 The output layer in the feature extraction model is only used to calculate the loss function L during training. y , the output layer will be discarded after the feature extraction model training is completed.

[0142] S2033. Construct a cross-modal loss function based on the second context feature vector of each training data and the transcribed text of the speech data in each training data.

[0143] Optionally, the cross-modal loss function can use the CTC (Connectionist Temporal Classification) loss function That is, the second context feature vector from the training data Restore the unmasked character sequence Y t (i.e., the transcribed text of the speech data in the training data).

[0144] S2034. Determine the target loss function as the weighted sum of the first loss function, the second loss function, and the cross-modal loss function.

[0145] Specifically, the weighted sum of the first loss function, the second loss function, and the cross-modal loss function can be expressed by the following formula:

[0146] L ctx =αL s +βL y +(1-α-β)L m

[0147] Among them, α and β (0<α, β<1) are weight coefficients used to control the effects of different loss functions.

[0148] S2035. According to the target loss function, adjust the parameters of the feature extraction model through back propagation.

[0149] Optionally, at least one of the speech encoder and the text encoder is a pre-trained model.

[0150] It is understandable that if the speech encoder and the text encoder are pre-trained models, back propagation adjusts the parameters of the feature extraction model, specifically adjusts the parameters of the cross-modal encoder.

[0151] Optionally, the training stop condition can be Lctx When the loss value drops to the first value and the first value no longer changes, the training is stopped.

[0152] It should be noted that Figure 3 The training process of the feature extraction model is shown. When the feature extraction model training is completed, the trained feature extraction model is used to Figure 2 In the training of the speech recognition model shown, the parameters of the feature extraction model remain unchanged during speech recognition model training; they are only used to extract contextual feature vectors to assist in recognition model training. Furthermore, during speech recognition model training and actual use, the text encoder branch and all masking operations in the feature extraction model are not performed (and are discarded). Only the speech encoder and cross-modal encoder are retained, and the contextual feature vectors are integrated into the encoder portion of the recognition model through an attention mechanism.

[0153] The speech recognition model training method provided in this embodiment obtains a second training sample set comprising multiple training data during any iteration of feature extraction model training, each training data comprising speech data and a transcribed text of the speech data. For each training data in the second training sample set, the training data is used as input to the feature extraction model, and a first context feature vector of the training data is output. The training data after modal masking is used as input to the feature extraction model, and a second context feature vector of the training data is output. The parameters of the feature extraction model are adjusted based on the first context feature vector of each training data and the second context feature vector of each training data in the second training sample set until a stop training condition is met. The feature extraction model determined by the iteration process that meets the stop training condition is determined as the trained feature extraction model. Through the training of the feature extraction model, the feature extraction model can directly extract the context feature vector of the speech data, as well as extract the context feature vector of the historical speech data from the historical speech data of the speech data. This avoids the problem in the prior art of inputting the recognition results of the historical speech into the language model to extract context features, which has the problem of low speech recognition accuracy due to accumulated recognition errors. The recognition accuracy of entities in specific domains can also be improved.

[0154] The present application also provides a speech recognition model training method, which can adopt two-stage training: the first stage can specifically adopt Figure 4 The training method shown in the figure first trains the feature extraction model, fixes the model parameters after obtaining the feature extraction model, and then uses Figure 2 The training method shown is used to train the speech recognition model. In this stage, the model parameters of the feature extraction model are not updated, and are only used to extract context feature vectors to assist in the recognition model training. For the specific process, please refer to Figure 2 and Figure 4The description in the illustrated embodiment will not be repeated here.

[0155] Figure 7 This is a flow chart of a speech recognition method provided in an embodiment of the present application. The execution subject of this method can be a terminal device or other computer device, such as Figure 7 As shown, the method may include:

[0156] S401: Acquire a voice signal.

[0157] S402: Input the speech signal and the historical speech signal of the speech signal into a pre-trained speech recognition model, and output the speech recognition result of the speech signal.

[0158] Among them, the speech recognition model is based on Figure 2 The method shown is trained.

[0159] Furthermore, the speech recognition model includes a feature extraction model and a recognition model, which inputs the speech signal and the historical speech signal of the speech signal into the pre-trained speech recognition model and outputs the speech recognition result of the speech signal, which may specifically include:

[0160] S4021. Input the speech signal and the historical speech signal of the speech signal into a feature extraction model, and output a context feature vector of the speech signal.

[0161] S4022: Input the speech signal and the context feature vector of the speech signal into the recognition model, and output the speech recognition result of the speech signal.

[0162] Figure 8 A schematic diagram of the structure of a speech recognition model provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the speech recognition model includes a feature extraction model and a recognition model, with the speech signal as S t , the historical speech signal of the speech signal is S t-1 For example, the speech signal S t and the historical speech signal S t-1 Input feature extraction model, output context feature vector H of speech signal t and historical speech signal S t-1 The context feature vector H t-1 , then the speech signal S t , the context feature vector H of the speech signal t And the historical speech signal S t-1 The context feature vector H t-1 Input recognition model, output speech signal S t The speech recognition result Y t * .

[0163] The speech recognition method provided in this embodiment adopts a speech recognition model including a feature extraction model and a recognition model. When performing speech recognition, the context feature vector of the speech signal is extracted based on the speech signal and the historical speech signal of the speech signal, and then the speech recognition result is obtained based on the speech signal and the context feature vector of the speech signal. Since the context information of the speech signal is combined during speech recognition, the accuracy of speech recognition can be improved, especially the recognition accuracy of long-tail entities in long speech recognition scenarios, that is, the recognition accuracy of entities in specific fields can be improved.

[0164] For example, the present embodiment of the present application uses two test sets to compare the recognition error rates of the speech recognition method provided by the present embodiment of the present application with those of the existing speech recognition method based on the long-context language model method, as shown in Table 1 below:

[0165] Table 1 Comparison of word recognition error rates

[0166]

[0167] As can be seen from Table 1, the lower the recognition word error rate, the better the speech recognition performance. The speech recognition method provided in the embodiment of the present application has a high recognition accuracy.

[0168] Figure 9 A schematic diagram of the structure of a speech recognition model training device provided in an embodiment of the present application is shown in FIG. Figure 9 As shown, the device may include: an acquisition module 11, a first processing module 12, a second processing module 13 and a parameter adjustment module 14, wherein:

[0169] The acquisition module 11 is used to acquire a first training sample set during any iteration of speech recognition model training, where the first training sample set includes multiple training data, each training data includes speech data, a transcribed text of the speech data, and historical speech data of the speech data, and the speech recognition model includes a feature extraction model and a recognition model;

[0170] The first processing module 12 is configured to, for each training data in the first training sample set, use the speech data in the training data as input to a feature extraction model and output a context feature vector of the speech data, and use the historical speech data of the speech data as input to the feature extraction model and output a context feature vector of the historical speech data, wherein the feature extraction model is obtained through pre-training;

[0171] The second processing module 13 is used to use the speech data, the context feature vector of the speech data and the context feature vector of the historical speech data as inputs of the recognition model, and output the recognition text of the speech data;

[0172] The parameter adjustment module 14 is used to adjust the parameters of the recognition model according to the recognition text of each speech data and the transcription text of each speech data in the first training sample set obtained in each iterative process until the training stop condition is met, and the speech recognition model determined by the iterative process that meets the training stop condition is determined as the trained speech recognition model.

[0173] Optionally, the acquisition module 11 is further configured to:

[0174] During any iteration of the feature extraction model training, a second training sample set is obtained, where the second training sample set includes a plurality of training data, each training data including speech data and a transcribed text of the speech data;

[0175] The first processing module 12 is also used to: for each training data in the second training sample set, use the training data as the input of the feature extraction model, output the first context feature vector of the training data, use the training data after modal masking processing as the input of the feature extraction model, output the second context feature vector of the training data.

[0176] The parameter adjustment module 14 is also used to adjust the parameters of the feature extraction model according to the first context feature vector of each training data and the second context feature vector of each training data in the second training sample set obtained in each iterative process until the training stop condition is met, and the feature extraction model determined by the iterative process that meets the training stop condition is determined as the trained feature extraction model.

[0177] Optionally, the feature extraction model includes a speech encoder, a text encoder, and a cross-modal encoder.

[0178] Optionally, the first processing module 12 is specifically configured to: use the speech data in the training data as input to the speech encoder and output a first speech feature vector of the training data;

[0179] Taking the transcribed text of the speech data as input of the text encoder, and outputting the first text feature vector of the training data;

[0180] The first speech feature vector of the training data and the first text feature vector of the training data are used as inputs of a cross-modal encoder, and the first context feature vector of the training data is output. The cross-modal encoder is used to associate the first speech feature vector of the training data with the first text feature vector of the training data.

[0181] Optionally, the training data after modality masking processing includes speech data after modality masking processing and transcribed text data after modality masking processing. Before the first processing module 12 uses the training data after modality masking processing as input to the feature extraction model and outputs the second context feature vector of the training data, it is further configured to:

[0182] Performing frame random masking on the speech feature sequence corresponding to the speech data in the training data to obtain a frame-masked speech feature sequence;

[0183] The product of the masked speech feature sequence and the speech modality mask is determined as the speech data after modality masking processing;

[0184] Performing random character masking on a character sequence corresponding to the transcribed text of the speech data to obtain a masked character sequence;

[0185] The product of the masked character sequence and the modal mask of the training data transcript is determined as the transcribed text data after modal masking processing.

[0186] Optionally, the first processing module 12 is specifically configured to:

[0187] Using the speech data after modal masking as input to a speech encoder, and outputting a second speech feature vector of the training data;

[0188] The transcribed text data after modality masking is used as input of a text encoder, and a second text feature vector of the training data is output;

[0189] The second speech feature vector of the training data and the second text feature vector of the training data are used as inputs of a cross-modal encoder, and a second context feature vector of the training data is output. The cross-modal encoder is used to associate the second speech feature vector of the training data with the second text feature vector of the training data.

[0190] Optionally, the parameter adjustment module 14 is specifically configured to:

[0191] Constructing a first loss function based on the first context feature vector of each training data and the second context feature vector of each training data in the second training sample set;

[0192] Constructing a second loss function based on the masked characters in the character sequence corresponding to the transcribed text of the speech data in each training data and the masked characters in the character sequence predicted by the feature extraction model, where the masked characters in the predicted character sequence are the output of the second context feature vector of the training data through the output layer;

[0193] Constructing a cross-modal loss function based on the second context feature vector of each training data and the transcribed text of the speech data in each training data;

[0194] The weighted sum of the first loss function, the second loss function and the cross-modal loss function is determined as the target loss function;

[0195] According to the target loss function, backpropagation adjusts the parameters of the feature extraction model.

[0196] Optionally, at least one of the speech encoder and the text encoder is a pre-trained model.

[0197] Optionally, the recognition model includes an encoder and a decoder, and the second processing module 13 is specifically configured to:

[0198] Taking speech data as input of the encoder, the speech feature vector of the speech data is output;

[0199] The decoder takes the speech feature vector of the speech data, the context feature vector of the speech data and the context feature vector of the historical speech data as input and outputs the recognition text of the speech data. The decoder is used to predict the recognition text of the speech data based on the speech feature vector of the speech data, the context feature vector of the speech data and the context feature vector of the historical speech data through the attention mechanism.

[0200] Optionally, the parameter adjustment module 14 is specifically configured to:

[0201] Constructing a loss function based on the recognized text of each speech data and the transcribed text of each speech data in the first training sample set;

[0202] According to the loss function, backpropagation adjusts the parameters of the recognition model.

[0203] Figure 10 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application is shown in FIG. Figure 10 As shown, the device may include: an acquisition module 21 and a speech recognition model 22, wherein:

[0204] The acquisition module 21 is used to acquire the voice signal;

[0205] The speech recognition model 22 is used to input the speech signal and the historical speech signal of the speech signal into the pre-trained speech recognition model, and output the speech recognition result of the speech signal. The speech recognition model is based on Figure 2 The method shown is trained.

[0206] Optionally, the speech recognition model 22 is used to: input the speech signal and the historical speech signal of the speech signal into a feature extraction model, and output a context feature vector of the speech signal and a context feature vector of the historical speech signal;

[0207] The context feature vector of the speech signal and the context feature vector of the historical speech signal are input into the recognition model, and the speech recognition result of the speech signal is output.

[0208] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 8 The speech recognition model training device shown or Figure 9 The voice recognition device shown can execute the method embodiment corresponding to the computer device, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiment corresponding to the computer device, which will not be repeated here for the sake of brevity.

[0209] The speech recognition model training device and speech recognition device of the embodiment of the present application are described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0210] Figure 11 It is a schematic block diagram of a computer device 700 provided in an embodiment of the present application.

[0211] like Figure 11 As shown, the computer device 700 may include:

[0212] The memory 710 and the processor 720 are configured to store computer programs and transmit the program code to the processor 720. In other words, the processor 720 can call and run the computer program from the memory 710 to implement the method in the embodiment of the present application.

[0213] For example, the processor 720 may be configured to execute the above method embodiments according to instructions in the computer program.

[0214] In some embodiments of the present application, the processor 720 may include but is not limited to:

[0215] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0216] In some embodiments of the present application, the memory 710 includes but is not limited to:

[0217] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0218] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 710 and executed by the processor 720 to implement the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0219] like Figure 11 As shown, the computer device may also include:

[0220] The transceiver 730 may be connected to the processor 720 or the memory 710 .

[0221] The processor 720 may control the transceiver 730 to communicate with other devices. Specifically, the processor 720 may send information or data to other devices or receive information or data sent by other devices. The transceiver 730 may include a transmitter and a receiver. The transceiver 730 may further include one or more antennas.

[0222] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0223] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. In other words, the present application also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.

[0224] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0225] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0227] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0228] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A speech recognition model training method, characterized in that: The method comprises: During any iteration of speech recognition model training, a first training sample set is obtained, where the first training sample set includes a plurality of training data, each training data includes speech data, a transcribed text of the speech data, and historical speech data of the speech data, and the speech recognition model includes a feature extraction model and a recognition model; For each training data in the first training sample set, the speech data in the training data is used as the input of the feature extraction model, and the context feature vector of the speech data is output; the historical speech data of the speech data is used as the input of the feature extraction model, and the context feature vector of the historical speech data is output; the feature extraction model is obtained through pre-training, and the feature extraction model includes a speech encoder, a text encoder, and a cross-modal encoder; the context feature vector is a context-related high-level information representation; the context feature vector of the speech data is the output obtained by inputting the speech feature vector of the speech data and the text feature vector of the speech data into the cross-modal encoder; the context feature vector of the historical speech data is the output obtained by inputting the speech feature vector of the historical speech data and the text feature vector of the historical speech data into the cross-modal encoder; Taking the speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data as inputs of the recognition model, and outputting a recognition text of the speech data; According to the recognition text of each speech data in the first training sample set and the transcribed text of each speech data obtained in each iterative process, the parameters of the recognition model are adjusted until the training stop condition is met, and the speech recognition model determined by the iterative process that meets the training stop condition is determined as the trained speech recognition model.

2. The method according to claim 1, characterized in that The method further comprises: During any iteration of the feature extraction model training, a second training sample set is obtained, where the second training sample set includes a plurality of training data, each training data including speech data and a transcribed text of the speech data; For each training data in the second training sample set, using the training data as input to the feature extraction model, outputting a first context feature vector of the training data, and using the training data after modality masking processing as input to the feature extraction model, outputting a second context feature vector of the training data; According to the first context feature vector of each training data in the second training sample set and the second context feature vector of each training data obtained in each iterative process, the parameters of the feature extraction model are adjusted until the training stop condition is met, and the feature extraction model determined by the iterative process that meets the training stop condition is determined as the trained feature extraction model.

3. The method according to claim 2, characterized in that The step of taking the training data as input of the feature extraction model and outputting a first context feature vector of the training data includes: Taking speech data in the training data as input to the speech encoder, and outputting a first speech feature vector of the training data; Taking the transcribed text of the speech data as input to the text encoder, and outputting a first text feature vector of the training data; The cross-modal encoder takes the first speech feature vector of the training data and the first text feature vector of the training data as input and outputs the first context feature vector of the training data. The cross-modal encoder is used to associate the first speech feature vector of the training data with the first text feature vector of the training data.

4. The method according to claim 3, characterized in that The training data after the modality masking process includes speech data after the modality masking process and transcribed text data after the modality masking process. The training data after the modality masking process is used as input to the feature extraction model. Before outputting the second context feature vector of the training data, the method further includes: Performing frame random masking on a speech feature sequence corresponding to speech data in the training data to obtain a frame-masked speech feature sequence; Determining the product of the masked speech feature sequence and the speech modality mask as the speech data after the modality masking process; Performing random character masking on a character sequence corresponding to the transcribed text of the speech data to obtain a masked character sequence; The product of the masked character sequence and the text modality mask is determined as the transcribed text data after the modality masking process.

5. The method according to claim 4, characterized in that The method of using the training data after the modality masking process as input to the feature extraction model and outputting a second context feature vector of the training data includes: Using the speech data after the modal masking process as input to the speech encoder, and outputting a second speech feature vector of the training data; Taking the transcribed text data after the modality masking process as input to the text encoder, and outputting a second text feature vector of the training data; The cross-modal encoder takes the second speech feature vector of the training data and the second text feature vector of the training data as input and outputs the second context feature vector of the training data. The cross-modal encoder is used to associate the second speech feature vector of the training data with the second text feature vector of the training data.

6. The method according to claim 2, characterized in that The adjusting the parameters of the feature extraction model according to the first context feature vector of each training data and the second context feature vector of each training data in the second training sample set includes: Constructing a first loss function according to the first context feature vector of each training data in the second training sample set and the second context feature vector of each training data; Constructing a second loss function based on the masked characters in the character sequence corresponding to the transcribed text of the speech data in each training data and the masked characters in the character sequence predicted by the feature extraction model, wherein the masked characters in the predicted character sequence are outputs of the second context feature vector of the training data through the output layer; Constructing a cross-modal loss function based on the second context feature vector of each training data and the transcribed text of the speech data in each training data; Determine a target loss function as a weighted sum of the first loss function, the second loss function, and the cross-modal loss function; According to the target loss function, back propagation is used to adjust the parameters of the feature extraction model.

7. The method according to claim 1, characterized in that At least one of the speech encoder and the text encoder is a pre-trained model.

8. The method according to claim 1, characterized in that The recognition model includes an encoder and a decoder. The recognition model takes the speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data as inputs and outputs the recognized text of the speech data, including: Taking the speech data as input of the encoder, and outputting a speech feature vector of the speech data; The decoder takes the speech feature vector of the speech data, the context feature vector of the speech data and the context feature vector of the historical speech data as input and outputs the recognition text of the speech data. The decoder is used to predict the recognition text of the speech data based on the speech feature vector of the speech data, the context feature vector of the speech data and the context feature vector of the historical speech data through an attention mechanism.

9. The method according to claim 1, characterized in that The adjusting the parameters of the recognition model according to the recognition text of each speech data and the transcribed text of each speech data in the first training sample set includes: Constructing a loss function based on the recognized text of each speech data and the transcribed text of each speech data in the first training sample set; According to the loss function, back propagation is used to adjust the parameters of the recognition model.

10. A speech recognition method, characterized in that: include: Acquire voice signals; The speech signal and the historical speech signal of the speech signal are input into a pre-trained speech recognition model, and a speech recognition result of the speech signal is output, wherein the speech recognition model is trained according to the method according to any one of claims 1-9.

11. The method according to claim 10, characterized in that The speech recognition model includes a feature extraction model and a recognition model. The step of inputting the speech signal and the historical speech signal of the speech signal into the pre-trained speech recognition model and outputting the speech recognition result of the speech signal includes: Inputting the speech signal and the historical speech signal of the speech signal into the feature extraction model, and outputting the context feature vector of the speech signal and the context feature vector of the historical speech signal; The context feature vector of the speech signal and the context feature vector of the historical speech signal are input into the recognition model, and a speech recognition result of the speech signal is output.

12. A speech recognition model training device, characterized in that: include: an acquisition module, configured to acquire a first training sample set during any iteration of speech recognition model training, wherein the first training sample set includes a plurality of training data, each training data including speech data, a transcribed text of the speech data, and historical speech data of the speech data, and the speech recognition model includes a feature extraction model and a recognition model; A first processing module is configured to, for each training data in the first training sample set, use the speech data in the training data as input to the feature extraction model to output a context feature vector of the speech data, and use the historical speech data of the speech data as input to the feature extraction model to output a context feature vector of the historical speech data, wherein the feature extraction model is obtained through pre-training, the feature extraction model includes a speech encoder, a text encoder, and a cross-modal encoder, the context feature vector is a high-level information representation related to context, the context feature vector of the speech data is an output obtained by inputting the speech feature vector of the speech data and the text feature vector of the speech data into the cross-modal encoder, and the context feature vector of the historical speech data is an output obtained by inputting the speech feature vector of the historical speech data and the text feature vector of the historical speech data into the cross-modal encoder; a second processing module, configured to use the speech data, the context feature vector of the speech data, and the context feature vector of the historical speech data as inputs of the recognition model, and output a recognized text of the speech data; A parameter adjustment module is used to adjust the parameters of the recognition model according to the recognition text of each speech data in the first training sample set and the transcription text of each speech data obtained in each iterative process until the training stop condition is met, and the speech recognition model determined by the iterative process that meets the training stop condition is determined as the trained speech recognition model.

13. A speech recognition device, characterized in that: include: An acquisition module, used for acquiring a voice signal; A speech recognition module is used to input the speech signal and the historical speech signal of the speech signal into a pre-trained speech recognition model, and output a speech recognition result of the speech signal, wherein the speech recognition model is trained according to the method according to any one of claims 1-9.

14. A computer device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 9 or 10 to 11.

15. A computer-readable storage medium, characterized in that Comprising instructions which, when run on a computer program, cause the computer to perform the method of any one of claims 1 to 9 or 10 to 11.

16. A computer program product comprising instructions, characterized in that When the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 9 or 10 to 11.

Images