Speech recognition model training method, speech recognition method and related device

Abstract

The invention provides a speech recognition model training method, a speech recognition method and a related device. The speech recognition model training method comprises the following steps: inputting training data into a speech recognition model for first processing to obtain a first vector, a second vector and a third vector; wherein the first processing comprises the following steps: encoding a first sample voice to obtain a first vector; performing feature fusion on the first vector and the first word through a first module to obtain a second vector; decoding the first vector and the second vector to obtain a third vector; determining a first loss value based on the third vector; and updating parameters of a first module in the speech recognition model based on the first loss value. According to the invention, the recognition accuracy of the speech recognition model in specific types of vocabularies can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech recognition model training method, a speech recognition method and related devices. Background Art

[0002] With the rapid development of artificial intelligence technology, speech recognition technology has become one of the bridges connecting humans and the digital world. The application scope of speech recognition technology is expanding and penetrating into all aspects of life: from smart homes, car navigation systems, smart assistants to medical health, customer service and even education and entertainment. With its unique interactive method, speech recognition technology provides users with a more natural and convenient user experience.

[0003] However, in the related art, due to the limitation of the training mechanism of the speech recognition model, the performance of the speech recognition model is heavily dependent on the training data. When the training data does not cover certain specific types of words, it will be difficult for the speech recognition model to accurately capture and understand the context and meaning of these words, resulting in a decrease in the accuracy of speech recognition for specific words. If a certain amount of training data containing specific types of words is added to the training set, it is possible that the diversity of the training data will be unbalanced, and the complexity of speech in the real world cannot be fully reflected, resulting in a decrease in the accuracy of speech recognition. Therefore, the speech recognition model in the related art has the problem of low recognition accuracy, especially the problem of low recognition accuracy for specific types of words. Summary of the invention

[0004] The embodiments of the present application provide a speech recognition model training method, a speech recognition method and related devices, which can improve the recognition accuracy of the speech recognition model on specific types of vocabulary.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a speech recognition model training method, the method comprising: inputting training data into the speech recognition model for first processing to obtain a first vector, a second vector and a third vector; wherein the speech recognition model comprises a first module, the training data comprises a first sample speech and a sample text, the sample text comprises a first word belonging to a first type; the first processing comprises: encoding the first sample speech to obtain the first vector; performing feature fusion of the first vector and the first word through the first module to obtain the second vector; decoding the first vector and the second vector to obtain the third vector; determining a first loss value based on the third vector; and updating the parameters of the first module in the speech recognition model based on the first loss value.

[0007] An embodiment of the present application provides a speech recognition method, the method comprising: inputting first data into a speech recognition model for third processing to obtain a ninth vector, a tenth vector and an eleventh vector; wherein the speech recognition model comprises an encoding module, a first module and a decoding module, the first data comprises a first speech and a second text, the second text comprises a fourth word belonging to a second type; the third processing comprises: encoding the first speech by the encoding module to obtain the ninth vector; performing feature fusion on the ninth vector and the fourth word by the first module to obtain the tenth vector; decoding the ninth vector and the tenth vector by the decoding module to obtain the eleventh vector; and determining the words belonging to the second type in the first speech based on the eleventh vector.

[0008] An embodiment of the present application provides a speech recognition model training device, comprising: a first data processing module, used to input training data into a speech recognition model for first processing to obtain a first vector, a second vector and a third vector; wherein the speech recognition model comprises a first module, and the training data comprises a first sample speech and a sample text, and the sample text comprises a first word belonging to a first type; the first processing comprises: encoding the first sample speech to obtain the first vector; performing feature fusion of the first vector and the first word through the first module to obtain the second vector; decoding the first vector and the second vector to obtain the third vector; a loss value determination module, used to determine a first loss value based on the third vector; and a parameter updating module, used to update the parameters of the first module in the speech recognition model based on the first loss value.

[0009] An embodiment of the present application provides a speech recognition device, including: a second data processing module, used to input first data into a speech recognition model for third processing to obtain a ninth vector, a tenth vector and an eleventh vector; wherein the speech recognition model includes an encoding module, a first module and a decoding module, the first data includes a first speech and a second text, and the second text includes a fourth word belonging to the second type; the third processing includes: encoding the first speech by the encoding module to obtain the ninth vector; performing feature fusion on the ninth vector and the fourth word by the first module to obtain the tenth vector; decoding the ninth vector and the tenth vector by the decoding module to obtain the eleventh vector; a determination module, used to determine the words belonging to the second type in the first speech based on the eleventh vector.

[0010] An embodiment of the present application provides an electronic device, which includes: a memory for storing computer-executable instructions; and a processor for executing the computer-executable instructions stored in the memory to implement the speech recognition model training method and speech recognition method provided in the embodiment of the present application.

[0011] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the speech recognition model training method and speech recognition method provided in the embodiment of the present application when executed by a processor.

[0012] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the speech recognition model training method and speech recognition method provided in the embodiment of the present application are implemented.

[0013] The embodiments of the present application have the following beneficial effects:

[0014] Since the first loss value used to update the parameters of the first module is determined based on the third vector, and the third vector is obtained after decoding the first vector and the second vector, that is, the third vector fuses the information corresponding to the first vector and the second vector, and the first vector is obtained by encoding the first sample speech, and the second vector is obtained by feature fusion of the first vector and the first word, therefore, the first vector is a vector that can characterize the speech features of the first sample speech, and the second vector is a vector that fuses the speech features of the first sample speech and the word features of the first word, then the third vector will also be a vector that fuses the speech features of the first sample speech and the word features of the first word. In this way, the first loss value determined based on the third vector can be used to evaluate the effect of the fusion of the speech features of the first sample speech and the word features of the first word, so that after updating the parameters of the first module based on the first loss value, the trained speech recognition model can more accurately fuse the speech features and the word features, thereby improving the recognition accuracy of the speech recognition model for the first type of words, that is, the speech recognition model trained by the method of the embodiment of the present application can improve the recognition accuracy on a specific type of vocabulary. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A It is a flowchart of the speech recognition model training method provided in the embodiment of the present application;

[0016] Figure 1B is a schematic diagram of a process for determining a second vector provided in an embodiment of the present application;

[0017] Figure 1C is a schematic diagram of a process for determining a third vector provided in an embodiment of the present application;

[0018] Figure 1D is a schematic diagram of a process for determining a fourth loss value and a fifth loss value provided in an embodiment of the present application;

[0019] Figure 1E It is a schematic diagram of a process for determining a sample text provided in an embodiment of the present application;

[0020] Figure 1F It is a schematic diagram of a process for updating parameters of an encoding module and a decoding module provided in an embodiment of the present application;

[0021] Figure 1G It is a flowchart of the speech recognition method provided in the embodiment of the present application;

[0022] Figure 2A is a schematic diagram of the structure of the initial speech recognition model provided in the embodiment of the present application;

[0023] Figure 2B is a schematic diagram of the structure of a speech recognition model to be trained provided in an embodiment of the present application;

[0024] Figure 2C is a structural diagram of the first module provided in an embodiment of the present application;

[0025] Figure 2D It is a structural diagram of the trained speech recognition model provided in the embodiment of the present application when in use;

[0026] Figure 2E It is a structural schematic diagram of a robot voice outbound calling device provided in an embodiment of the present application;

[0027] Figure 3 It is a structural schematic diagram of a speech recognition model training device provided in an embodiment of the present application;

[0028] Figure 4 is a structural diagram of a speech recognition device provided in an embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of the composition structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] If similar descriptions of "first / second / third" appear in the application documents, the following instructions are added. In the following description, the terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0034] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0036] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0037] 1) Large Language Models (LLMs) refer to a type of natural language processing model based on deep learning. Large language models learn the statistical characteristics and laws of language by training a large amount of text data, so that they can understand and generate natural language text. Large language models usually have billions to hundreds of billions of parameters and can capture complex relationships and patterns in language text.

[0038] 2) Prompt: refers to a piece of text input to the large language model to guide the large language model to generate corresponding output. Prompts are usually used to stimulate the large language model to express specific topics or tasks. Prompts are a way for humans to interact with large language models and can be regarded as "prompts" or "inquiries" to the large language model.

[0039] 3) Network parameters refer to the variables inside the model network, which are adjusted by the learning algorithm during the training process so that the model network can perform specific tasks more accurately, such as weights and biases in the model network.

[0040] 4) Loss Value is a measure of the difference between the model's predictions and the true results. The loss function is used to calculate the loss value, which is a core component of the machine learning model optimization process. The purpose of the loss value is to guide the learning process by quantifying the model's errors so that the model network can reduce these errors by adjusting its parameters.

[0041] 5) Connectionist Temporal Classification (CTC) loss is a metric used to evaluate the performance of sequence-to-sequence mapping models (such as speech recognition, optical character recognition, etc.). CTC loss is based on the connectionist temporal classification algorithm and is used to deal with problems where the input and output sequences are of different lengths and do not require pre-alignment.

[0042] 6) Label Smoothing Cross Entropy loss value is the loss value obtained by the label smoothing cross entropy function. The label smoothing cross entropy function is an optimization of the traditional cross entropy loss function, which aims to reduce the overfitting phenomenon of the model and improve the generalization ability of the model. The calculation method of the label smoothing cross entropy loss value is similar to the traditional cross entropy loss, but it uses a smoothed label distribution. In this way, the model can be more cautious in predicting each category and will not be too confident, which helps the model generalize and reduce the risk of overfitting.

[0043] 7) Attention Mechanism is a deep learning technology that simulates human attention behavior. The attention mechanism allows the model to allocate limited computing resources to different parts of the input when processing input information, thereby focusing on the most relevant information. By assigning different weights to different parts of the input, the attention mechanism can help the model process high-dimensional data such as images, text or speech more effectively, improving the performance and efficiency of the model.

[0044] 8) Self-Attention is an attention mechanism widely used in deep learning models, especially in natural language processing tasks. The self-attention mechanism allows the model to "pay attention to itself" for each element in the input vector (such as a word or character) and calculate the correlation score with other elements. In this way, the self-attention mechanism is able to capture the internal dependencies in the input vector and generate representations of each element that incorporate the information of all elements in the vector. The core of the self-attention mechanism is to determine the weight distribution between different elements in the input vector by calculating the dot product or similarity score between the query, key, and value. These weights are then used to generate a weighted sum, which is the output of the self-attention mechanism, reflecting the combined influence of all elements in the input vector on the current element.

[0045] 9) Multi-Head Attention Network is a deep learning network structure that captures multiple key information at different positions in the input vector by using multiple self-attention mechanisms in parallel. Each "head" (attention allocator) independently calculates the attention weight to focus on different aspects of the input, and then combines these attention results to produce the final output.

[0046] 10) Transformer network is a neural network architecture based on the self-attention mechanism in the field of deep learning, which is used to solve sequence-to-sequence tasks such as machine translation. It mainly consists of two parts: encoder and decoder. Each part contains multiple identical layers, and each layer contains a self-attention mechanism and a feedforward neural network.

[0047] 11) Convolutional Self-Attention Network (Conformer) is a deep learning network structure that combines the advantages of the self-attention mechanism of the Transformer network structure and the advantages of Convolutional Neural Networks (CNN), aiming to improve the performance of sequence data processing tasks such as speech recognition, natural language processing and machine translation. The core design goal of Conformer is to integrate the powerful long-distance dependency modeling ability of the Transformer network structure with the high efficiency of the Convolutional Neural Network in capturing local features.

[0048] 12) Training Data is the data set used to train the model. Training data usually contains multiple feature data and corresponding label data. The model predicts or classifies by learning the rules and patterns in the training data. The quality and quantity of training data have a crucial impact on the performance and accuracy of the model.

[0049] 13) Training batches refer to the subset of training data used by the model in one training iteration. In the actual training of machine learning models, especially deep learning models, due to memory limitations and computational efficiency considerations, all training data is usually not input into the model at one time for training. Instead, the data set is divided into multiple small subsets, which are called training batches.

[0050] 14) Labeled Data refers to data in which each sample (such as an image, a piece of text, or a transaction record) has a corresponding label or target value. This label or target value is defined in advance and is usually the result that the model is expected to predict. For example, in the image recognition task, the image and its corresponding category (such as "cat", "dog", etc.) are labeled data; in the email classification task, the email and its corresponding category (such as "spam", "non-spam") are also labeled data. Labeled data is the basis of supervised learning, because supervised learning algorithms need to use these labels to train the model so that it can make predictions on unlabeled data.

[0051] 15) Feature Data refers to the data used to describe all the information of each sample in the data set. In machine learning, feature data is the input variable used to predict the label. Features are attributes extracted from the data that are meaningful to the model, and features can represent the key information in the data. For example, in a house price prediction model, feature data may include the area of ​​the house, the number of rooms, the year of construction, etc.; in a recommendation system, feature data may include the user's historical purchase records, browsing history, ratings, etc. Selecting and constructing effective features is crucial to the performance of the model.

[0052] In the related art, the performance of the speech recognition model is heavily dependent on the training data behind it due to the limitations of the training mechanism of the speech recognition model. When the training data does not cover certain specific words sufficiently, it will be difficult for the speech recognition model to accurately capture and understand the context and meaning of these words, resulting in a decrease in the accuracy of speech recognition for specific words. If a certain amount of training data containing specific types of words is added to the training set, it is possible that the diversity of the training data will be unbalanced, and the complexity of speech in the real world will not be fully reflected, resulting in a decrease in the accuracy of speech recognition. Therefore, the speech recognition model in the related art has the problem of low recognition accuracy, especially the problem of low recognition accuracy for specific types of words.

[0053] Based on at least one of the above problems existing in the related art, the embodiment of the present application provides a speech recognition model training method, which processes the training data through the encoding module, the first module and the decoding module to obtain multiple vectors, and updates the network parameters in the first module based on the loss values ​​of these vectors, so that the speech recognition model can more comprehensively consider the relationship between speech and text; based on the third vector obtained by decoding the first vector and the second vector, the first loss value obtained by loss calculation can evaluate the effect of the fusion of speech features and word features of the first word, and by continuously optimizing the result of the first loss value, the speech recognition model can more accurately fuse the speech features and the word features of the first word, thereby improving the recognition accuracy of the first word. In summary, the embodiment of the present application optimizes the first module through the first loss value, so that the speech recognition model can learn more abundant information, thereby improving the overall recognition performance of the speech recognition model; on this basis, by continuously optimizing the result of the first loss value, it is helpful to improve the recognition accuracy of the speech recognition model on specific types of vocabulary.

[0054] See also Figure 1A , Figure 1A This is a flow chart of the speech recognition model training method provided in the embodiment of the present application. Figure 1A The steps shown are explained as Figure 1A As shown, the speech recognition model training method is described by taking the execution subject of the electronic device as an example, and the method includes the following steps 101 to 103:

[0055] Step 101: input the training data into the speech recognition model for first processing to obtain a first vector, a second vector and a third vector.

[0056] The speech recognition model includes at least a first module, and the training data includes a first sample speech and a sample text, and the sample text includes a first word belonging to a first type. The first sample speech is a piece of speech data in the sample data used in the speech recognition model training process. The sample text is the text data corresponding to the first sample speech, and it can also be understood that the sample text is a transcribed text of the first sample speech, that is, the content in the first sample speech is converted into text form, and the sample text is used to compare and associate with the speech data. The first word is a word of special concern in the sample text, and can also be called a target word. In an actual training scenario, the first word can be a word that needs to be specially recognized in a general field or a professional field for the current speech, such as a name, a place name, a brand name or other proper nouns.

[0057] The first processing includes: encoding the first sample speech to obtain a first vector; performing feature fusion on the first vector and the first word through the first module to obtain a second vector; and decoding the first vector and the second vector to obtain a third vector.

[0058] In some embodiments, the speech recognition model may include an encoding module, a first module and a decoding module. The first vector is obtained by encoding the first sample speech by the encoding module, the second vector is obtained by fusing the first vector and the first word by the first module, and the third vector is obtained by decoding the first vector and the second vector by the decoding module.

[0059] The encoding module is a module in the speech recognition model responsible for encoding the first sample speech, which can convert the first sample speech into a vector representation; a convolutional neural network (CNN) or a recurrent neural network (RNN) can be used as the encoding module to extract the speech signal features (vector representation) of the first sample speech, thereby obtaining the first vector. The first module is a module in the speech recognition model specifically used to process the first word of the first type, which can use an attention mechanism or a multi-head self-attention mechanism to fuse the encoded speech features with the text features of the first word, so as to better identify and associate the first word in the first sample speech. The decoding module is a module in the model responsible for decoding the encoded vector, which is used to generate the final semantic vector; structures such as a fully connected network (FCN) or a long short-term memory network (LSTM) can be used as the decoding module.

[0060] The first vector is a vector obtained by encoding the first sample speech through the encoding module. The first vector represents the speech signal characteristics of the first sample speech and is the basis for the speech recognition model to understand the speech content of the first sample speech. The second vector is a vector obtained by the first module after the first vector and the first word are fused. The second vector combines the speech characteristics of the first sample speech and the text characteristics of the first word, which helps the speech recognition model to more accurately identify and focus on the specific first word. The third vector is a vector obtained by decoding the first vector and the second vector through the decoding module. The third vector combines the overall speech signal characteristics of the first sample speech and the specific characteristics of the first word, providing the decoding module with richer information to generate a more accurate semantic vector.

[0061] Step 102: determine a third loss value based on the third vector.

[0062] In the embodiment of the present application, the first loss value is obtained by performing loss calculation based on the third vector.

[0063] In some embodiments, the third vector is obtained by decoding the first vector and the second vector through a decoding module. The third vector represents the final semantic understanding of the first sample speech by the speech recognition model. The third vector can be used to generate the final recognition result of speech recognition, that is, the final recognized text. The third loss value is a loss value calculated based on the third vector. The third loss value measures the accuracy of the final semantic understanding of the model. Moreover, since the third vector is obtained by decoding the first vector and the second vector, the third vector fuses the information corresponding to each of the first vector and the second vector, that is, the third vector is a vector that fuses the speech features of the first sample speech and the word features of the first word. Therefore, the first loss value obtained by loss calculation based on the third vector can be used to evaluate the effect of the fusion of the speech features of the first sample speech and the word features of the first word.

[0064] Loss calculation refers to the mathematical operation used to evaluate the difference between the text output by the speech recognition model and the sample text during the training process. The loss calculation method for calculating the first loss value can use the same loss calculation function or a different loss calculation function; for example, the CTC loss function, the mean squared error (MSE) loss function, the cross-entropy loss function, or the label smoothed cross-entropy loss can be used.

[0065] Step 103: Update the parameters of the first module in the speech recognition model based on the first loss value.

[0066] In some embodiments, the updating process of the network parameters in the first module can be implemented by a back propagation algorithm and a gradient descent method, etc. This process can be iterated until the first loss value reaches a preset value or satisfies a preset stop condition.

[0067] Through step 103, the first loss value is considered to perform parameter update, and by continuously optimizing the result of the first loss value, the trained speech recognition model can more accurately integrate speech features and word features, thereby improving the recognition accuracy of the speech recognition model for the first type of words.

[0068] In the embodiment of the present application, the speech recognition model when the network parameters in the first module meet the first preset training stop condition can be determined as the trained speech recognition model.

[0069] In some embodiments, the first preset training stop condition is a rule that sets the training process to stop in order to prevent overfitting or achieve expected performance indicators when training the speech recognition model. The first preset training stop condition can be set according to the performance indicators of the speech recognition model (such as loss value, accuracy, etc.). When these performance indicators reach a certain threshold or change trend, the training process will stop. The first preset training stop condition can be that the first loss value drops below a specific threshold.

[0070] For example, in the field of speech recognition, the training stop condition can be set to the first loss value no longer significantly decreasing during the training process, to ensure that the speech recognition model does not overfit the training data, while maintaining good performance in practical applications. In this way, it can be ensured that the trained speech recognition model is the model with the best performance under certain resource constraints, which can not only avoid overfitting, but also save computing resources and improve training efficiency.

[0071] In some embodiments, see Figure 1B The process of performing feature fusion to obtain the second vector can be implemented through steps 201A to 203A, which are described in detail below.

[0072] Step 201A, encode the first word to obtain a feature vector of the first word.

[0073] Here, the first word can be encoded by the first word encoding network of the first module to obtain the feature vector of the first word.

[0074] In some embodiments, the first word encoding network refers to a network structure in the first module for processing and encoding the first word in text form. The function of the first word encoding network is to convert the first word into a numerical form that can be understood and processed by the machine, that is, a feature vector; the first word encoding network can include a variety of network structures, such as a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU), or a transformer based on a self-attention mechanism. The feature vector of the first word is the result obtained after processing by the first word encoding network, and the feature vector of the first word is a vector representation of the first word.

[0075] Step 202A, performing multi-head attention processing based on the feature vector and the first vector to obtain multiple query vectors, multiple key vectors, and multiple value vectors.

[0076] Here, the feature vector of the first word can be used as the key vector and the value vector, and the first vector can be used as the query vector. Through the first word fusion network in the first module, the query vector, key vector and value vector corresponding to each attention distributor in the first word fusion network can be determined. In some embodiments, the query vector, key vector and value vector can be linearly transformed to obtain a single-headed query vector, a single-headed key vector and a single-headed value vector; wherein the single-headed query vector is the linear transformation result of the query vector, the single-headed value vector is the linear transformation result of the value vector, and the single-headed key vector is the linear transformation result of the key vector; the first word fusion network includes a preset number of attention distributors.

[0077] In some embodiments, the first word fusion network is part of the first module, and the first word fusion network can be a multi-head attention network; the first word fusion network can use a self-attention mechanism to fuse the feature vector of the first word with the encoded speech feature (i.e., the first vector), and the first word fusion network can enable the speech recognition model to learn the association between the first word and the first sample speech. The first word fusion network can also include an attention allocator, which is a component in the first word fusion network. The attention allocator is responsible for calculating the degree of match between the query vector and the key vector, and assigning weights to the value vector according to these matching degrees. Each attention allocator can be regarded as a decision-making unit of a focus point.

[0078] The query vector represents the context information that needs to be paid attention to or queried at present; the key vector is a vector used for similarity calculation with the query vector; and the value vector is the actual information associated with the key vector. The single-head query vector, the single-head key vector, and the single-head value vector are the linear transformation results of the query vector, the key vector, and the value vector, respectively. In the multi-head attention network, each attention distributor will generate its own independent single-head query vector, single-head key vector, and single-head value vector. These vectors of the attention distributor are obtained by performing different linear transformations on the vectors input into the first word fusion network. The linear transformation is a mathematical operation performed on the original vectors (query vector, key vector, and value vector) through a weight matrix. The purpose is to change the representation space of the vectors so that these vectors can interact in a richer feature space, thereby improving the model performance and enabling the speech recognition model to learn more complex feature representations. The above-mentioned "preset number" refers to the number of attention distributors pre-defined in the first word fusion network. The number of attention distributors can determine the complexity of the information that the first word fusion network can process. In practical applications, the number of attention distributors needs to be reasonably set according to the requirements of specific tasks and the situation of computing resources.

[0079] Step 203A: determine a second vector based on multiple query vectors, multiple key vectors, and multiple value vectors.

[0080] In the embodiment of the present application, the attention allocator in the first word fusion network in the first module can first determine the weighted value vector corresponding to the single-head value vector based on the single-head query vector and the single-head key vector. Then, the weighted value vectors determined by all the attention allocators are fused to obtain the second vector.

[0081] In some embodiments, the weighted value vector is calculated by an attention allocator, that is, a vector obtained by weighting the single-head value vector according to the degree of match between the single-head query vector and the single-head key vector; the weights in the weighted value vector reflect the degree of attention that the attention allocator pays to different parts of the input vector.

[0082] During the implementation process, the weighted value vectors calculated by multiple attention allocators can be merged into a single vector, namely the second vector; during the implementation process, all weighted value vectors can be summed or weighted averaged to obtain a new vector (second vector) that integrates the information of all attention allocators.

[0083] Through step 201A to step 203A, a comprehensive vector (second vector) combining the focus of multiple attention allocators can be obtained. The second vector is more comprehensive and accurate than the output of any single attention allocator, because the second vector contains information from different angles and levels, so that the speech recognition model can more accurately capture the correspondence between the first word and the first sample speech, thereby improving the accuracy and robustness of the speech recognition model in recognizing the first word.

[0084] In some embodiments, see Figure 1C , the process of decoding to obtain the third vector can be implemented through steps 201B to 202B, which are described in detail below.

[0085] Step 201B, using a first weight to process the first vector to obtain a first weighted vector, and using a second weight to process the second vector to obtain a second weighted vector; wherein there is a preset weight ratio between the first weight and the second weight.

[0086] In some embodiments, a first weight for vector adjustment of the first vector and a second weight for vector adjustment of the second vector can be determined according to a preset weight ratio. The preset weight ratio is a pre-set ratio for adjusting the importance of the first vector and the second vector in the vector fusion process. This weight ratio is adjustable, allowing adjustment according to actual conditions during the training or application of the speech recognition model. The first weight is the weight for vector adjustment of the first vector. The second weight is the weight for vector adjustment of the second vector. The sum of the first weight and the second weight should be 1 to ensure that the contributions of the two vectors can be correctly weighted in the subsequent vector adjustment process.

[0087] The first weighted vector is a vector obtained by adjusting the first vector by applying the first weight, and the second weighted vector is a vector obtained by adjusting the second vector by applying the second weight. That is, the first weighted vector is obtained by weighting the first vector by the first weight, and the second weighted vector is obtained by weighting the second vector by the second weight.

[0088] Step 202B: determine a third vector based on the first weighted vector and the second weighted vector.

[0089] Here, the vector sum of the first weighted vector and the second weighted vector may be determined to obtain a third vector. In some embodiments, the first weighted vector and the second weighted vector may be element-wise added, that is, elements at corresponding positions in the two vectors are added to obtain a new vector, namely, the third vector.

[0090] Through steps 201B to 202B, the first vector and the second vector are weightedly calculated through the adjustable preset weight ratio, so that the speech recognition model can flexibly adjust the influence of the second vector on the final recognition result (the degree of feature enhancement of the first word information) during the training or application process, which helps the speech recognition model to make more accurate judgments in complex scenarios. That is, by adjusting the adjustable preset weight ratio, the speech recognition model can be optimized according to specific needs to improve the speech recognition performance.

[0091] In some embodiments, in addition to determining the first loss value based on the third vector, the second loss value can also be determined based on the first vector, and the third loss value can be determined based on the second vector. Accordingly, when updating the parameters of the first module in the speech recognition model based on the first loss value, the parameters of the first module in the speech recognition model can be updated based on the first loss value, the second loss value and the third loss value. That is to say, in the embodiment of the present application, the loss values ​​at three different levels will be comprehensively considered for parameter updating, and by continuously optimizing the results of the first loss value, the second loss value and the third loss value, the encoding effect of the speech recognition model on the first sample speech, the effect of fusing speech features and first word features, and the accuracy of semantic understanding can be optimized. The speech recognition model can more comprehensively consider the relationship between speech and text, and can more accurately fuse the speech features of the first sample speech and the text features of the first word, thereby improving the recognition accuracy of the first word.

[0092] In the embodiment of the present application, the second loss value is a loss value calculated based on the first vector, and the second loss value reflects the gap between the encoding effect of the speech recognition model on the first sample speech and the real speech features. The third loss value is a loss value calculated based on the second vector, and the third loss value measures the effect of the speech recognition model fusing the speech features of the first sample speech and the text features of the first word, that is, the performance of the first module, that is, the third loss value reflects the recognition accuracy of the speech recognition model for the first word. In other words, based on the second vector fused with the speech features and the word features of the first word, the second loss value obtained by loss calculation can evaluate the effect of the fusion of the speech features and the word features of the first word. By continuously optimizing the result of the second loss value, the speech recognition model can more accurately fuse the speech features and the word features of the first word, and improve the recognition accuracy for the first word. The third vector is obtained by fusing the first vector and the second vector by the decoding module. The third vector represents the final semantic understanding of the speech recognition model for the first sample speech. The third vector can be used to generate the final recognition result of speech recognition, that is, the final recognized text. Therefore, the first loss value determined based on the third vector measures the accuracy of the final semantic understanding of the model.

[0093] In addition, the above-mentioned first preset training stop condition may also be that the first comprehensive loss value drops below a specific threshold value, where the first comprehensive loss value is calculated based on the first loss value, the second loss value and the third loss value, and may be the weighted sum of the first loss value, the second loss value and the third loss value.

[0094] In summary, the embodiment of the present application enables the trained speech recognition model to learn richer information through the joint optimization of multiple loss values, thereby improving the overall recognition performance of the trained speech recognition model.

[0095] In some embodiments, the first processing further includes: encoding the first sample speech through the encoding module to obtain a fourth vector; performing feature fusion on the fourth vector and the first word through the first module to obtain a fifth vector; decoding the fifth vector through the decoding module to obtain a sixth vector; further, a fourth loss value can be determined based on the fifth vector, and a fifth loss value can be determined based on the sixth vector, and the fourth loss value and the fifth loss value can be further combined to update the parameters of the first module in the first speech recognition model. That is, in some embodiments, in addition to calculating the first loss value, the second loss value and the third loss value, the fourth loss value and the fifth loss value can also be calculated, and the parameters of the first module in the speech recognition model are updated based on the first loss value, the second loss value, the third loss value, the fourth loss value and the fifth loss value.

[0096] See also Figure 1D The process of obtaining the fourth loss value and the fifth loss value can be implemented through steps 301 to 303. The details are described below.

[0097] Step 301, encode the first sample speech through the N-layer convolutional self-attention network of the encoding module to obtain a fourth vector; wherein the encoding module includes M-layer convolutional self-attention networks, and N is less than M.

[0098] Here, encoding the first sample speech through the encoding module actually encodes the first sample speech through the N-layer convolutional self-attention network of the encoding module to obtain the fourth vector. For example, the first sample speech can be encoded through the first N layers of convolutional self-attention network of the encoding module to obtain the fourth vector.

[0099] In some embodiments, the convolutional self-attention network is a network structure that combines a convolutional neural network and a self-attention mechanism. M represents the total number of layers of the convolutional self-attention network in the encoding module; N represents the number of layers of the convolutional self-attention network used in the encoding module to encode the sample speech to obtain the fourth vector; N is less than M because the encoding module only uses the first N layers of the convolutional self-attention network in the process of encoding the fourth vector, instead of all the M layers of the convolutional self-attention network; for example, M can be 12 and N can be 9. The fourth vector is the result obtained by encoding the sample speech through the first N layers of the convolutional self-attention network of the encoding module, and the fourth vector represents the feature representation of the first N layers of the convolutional self-attention network of the sample speech.

[0100] Step 302: The first module performs feature fusion on the fourth vector and the first word to obtain a fifth vector.

[0101] In some embodiments, the fifth vector is the result of feature fusion of the fourth vector and the feature vector of the first word by the first module; the fifth vector represents the speech recognition model's understanding of the first word, and the fifth vector combines the spatiotemporal features of the sample speech (the fourth vector) and the semantic features of the first word.

[0102] Step 303, determining a fourth loss value based on the fifth vector; and determining a fifth loss value based on the sixth vector.

[0103] The fourth loss value is obtained by calculating the loss based on the fifth vector, and the fifth loss value is obtained by calculating the loss based on the sixth vector; the sixth vector is obtained by decoding the fifth vector by the decoding module. In some embodiments, the fourth loss value is obtained by calculating the loss based on the fifth vector, and the fourth loss value is used to measure the difference between the fifth vector predicted by the speech recognition model and the feature vector of the sample text, and the fourth loss value reflects the accuracy of the speech recognition model in speech recognition after fusing the feature vector of the first word.

[0104] The sixth vector is obtained by decoding the fifth vector through the decoding module. The sixth vector is the result of decoding an intermediate output in the encoding process. The sixth vector represents the initial understanding of the first sample speech input by the speech recognition model. The fifth loss value is calculated based on the loss of the sixth vector. The fifth loss value is used to measure the difference between the sixth vector predicted by the speech recognition model and the feature vector of the sample text. The fifth loss value reflects the performance of the speech recognition model in the semantic understanding process.

[0105] In the embodiment of the present application, by using the first N layers of convolutional self-attention networks in the encoding module to encode the first sample speech, and performing loss calculation on the output vector (i.e., the fourth vector) of the first N layers of convolutional self-attention networks, the information in the encoding module can be fully utilized, especially the speech features captured in the first N layers of convolutional self-attention networks. The whole process is concise and helps to improve the recognition effect of the speech recognition model; by increasing the loss calculation of the first N layers of convolutional self-attention networks (obtaining the fourth loss value and the fifth loss value), the information of the first N layers of convolutional self-attention networks is effectively transmitted to the last layer of the speech recognition model (that is, through back propagation based on the loss value, affecting the vector result obtained by the M-layer complete encoding module), which plays a role of high-speed mutual information transmission between the first N layers of convolutional self-attention networks and the last layer of the speech recognition model, so that the speech recognition model can learn useful information earlier in the training process, thereby accelerating the training speed and convergence effect.

[0106] In some embodiments, see Figure 1E (wherein A represents the above-mentioned steps 101 to 103). Before step 101, the method further includes steps 401 to 403, which are described in detail below.

[0107] Step 401: Perform speech recognition on a first sample speech to obtain a first text.

[0108] When performing speech recognition on the first sample speech, the acquired first sample speech can be input into the speech-to-text model to convert the first sample speech into the first text. In the implementation process, the first sample speech can be preprocessed first, and the preprocessing includes steps such as denoising and enhancement to improve the speech quality. Then, feature extraction is performed on the preprocessed first sample speech, and key features, such as Mel Frequency Cepstrum Coefficient (MFCC), are extracted from the preprocessed first sample speech. Then, speech decoding is performed on the key features, and the extracted key features are converted into the first text.

[0109] When performing speech recognition, the speech-to-text model can be a recurrent neural network (RNN) and its variants, such as a long short-term memory network (LSTM). These models can process time series data and are very effective for speech recognition.

[0110] Step 402: extracting second words belonging to the first type from the first text based on the preset prompt words.

[0111] Here, a large prediction module can be used to input a preset prompt word into the large prediction model for second word extraction. The preset prompt word is used to guide and instruct the large language model to extract the second word belonging to the first type from the first text. In the implementation process, the task of extracting the second word in the first text can be performed by the large language model of the preset prompt word.

[0112] Based on the preset prompt word, the second word belonging to the first type can be extracted from the first text. For example, for the first text of the customer service and the user, the preset prompt word can be set to "extract the user's occupation information from the text", then the large language model can extract the second word belonging to the user's occupation from the first text.

[0113] In the embodiment of the present application, a set of confusing words corresponding to the second word can also be determined, and then any one of the confusing words to be selected in the confusing word set is determined as the confusing word of the second word. The confusing word set includes a preset number of confusing words to be selected corresponding to any second word. The confusing word set is a set of words that are similar in pronunciation to the second word. The words in the confusing word set (the confusing words to be selected) may cause confusion in the speech recognition model when recognizing the first word. Therefore, these confusing words need to be considered in the training process to simulate the confusion of words with similar pronunciations that may occur in actual applications to improve the robustness of the speech recognition model. The "preset number" here is the number of predefined confusing words to be selected, and the number of confusing words to be selected determines the size of the confusing word set, and the number of confusing words to be selected can be set according to actual conditions.

[0114] In the embodiment of the present application, a large language model based on preset prompt words identifies second words in the first text and generates confused words similar to these second words. Compared with traditional manual labeling or simple rule matching methods, the accuracy and efficiency of confusing word extraction can be greatly improved, thereby improving the generalization ability of the speech recognition model in different scenarios.

[0115] Step 403: Replace a preset number of second words in the first text with third words to obtain a sample text.

[0116] The third word is a word with the same number of characters and similar pronunciation as the second word, that is, a confused word in the confused word set; the first word in the sample text includes the third word and the second word that has not been replaced. In the embodiment of the present application, part of the first word in the first text can be replaced with the confused word according to a preset ratio to obtain the sample text for training the speech recognition model.

[0117] In some embodiments, the preset ratio refers to a predefined ratio of confusing words used to replace the second person's words. This ratio of confusing words determines the number of confusing words that replace the second word in the sample text. In this embodiment of the application, this preset ratio can be set to 3%.

[0118] By replacing some confusing words with the second word through steps 401 to 403, it is possible to simulate the confusion of words with similar pronunciations that may occur in actual scenarios, so that the speech recognition model can learn the ability to distinguish these confusing words during the training process. That is, by adding such noise to the training data, the robustness and generalization ability of the speech recognition model in actual applications can be improved.

[0119] In some embodiments, see Figure 1F (wherein A represents the above-mentioned steps 101 to 103). Before step 101, the method further includes steps 501 to 503, which are described in detail below.

[0120] Step 501, input the second sample speech into the speech recognition model for second processing to obtain the seventh vector and the eighth vector; wherein the second processing includes: encoding the second sample speech by the encoding module to obtain the seventh vector; decoding the seventh vector by the decoding module to obtain the eighth vector.

[0121] Step 502, determining a sixth loss value based on the seventh vector; determining a seventh loss value based on the eighth vector.

[0122] Step 503: Based on the sixth loss value and the seventh loss value, update the parameters of the encoding module and the decoding module in the speech recognition model.

[0123] In an embodiment of the present application, before the speech recognition model is trained by the training data, that is, before the first module in the speech recognition model is trained by the training data, the encoding module and the decoding module in the speech recognition model may be trained first. At this time, for the speech recognition model to be trained, it may include an encoding module and a decoding module, the second sample speech may be input into the speech recognition model to be trained, the second sample speech may be encoded by the encoding module to obtain the seventh vector, and the seventh vector may be decoded by the decoding module to obtain the eighth vector. Thus, the sixth loss value is determined based on the seventh vector, and the seventh loss value is determined based on the eighth vector. Finally, based on the sixth loss value and the seventh loss value, the parameters of the encoding module and the decoding module in the speech recognition model are updated to obtain the trained encoding module and the trained decoding module. Afterwards, the first module is integrated between the trained encoding module and the trained decoding module to obtain the speech recognition model, and further the above steps 101 to 103 are used for training based on the training data.

[0124] In other embodiments, when training the encoding module and the decoding module in the speech recognition model to be trained, the input data includes the initial sample text in addition to the second sample speech; the seventh vector is obtained by encoding the second sample speech by the encoding module, and the eighth vector is obtained by decoding the seventh vector by the decoding module. In some embodiments, it can also include a twelfth vector and a thirteenth vector, the twelfth vector is obtained by encoding the second sample speech by the P-layer convolutional self-attention network of the encoding module, and the thirteenth vector is obtained by decoding the twelfth vector by the decoding module; the encoding module includes an M-layer convolutional self-attention network, and P is less than M.

[0125] In some embodiments, the second sample speech is a piece of speech data extracted from the initial training data; the initial sample text is the text content corresponding to the second sample speech; the initial training data should be speech recognition training data in a general field, which does not contain the first word. The speech recognition model to be trained is a model used for initial training during the training process of the speech recognition model; the speech recognition model to be trained includes an encoding module and a decoding module for processing the initial training data. The encoding module is responsible for converting the second sample speech into feature vectors, and the decoding module is responsible for converting these feature vectors into text output.

[0126] The seventh vector is a feature vector obtained by encoding the second sample speech through the encoding module, and the seventh vector represents the feature representation of the second sample speech. The twelfth vector is a feature vector obtained by encoding the second sample speech through the P-layer convolutional self-attention network of the encoding module, and the twelfth vector represents the feature information of the first N layers of convolutional self-attention network extracted from the second sample speech. P can be the same as N.

[0127] The eighth vector is a feature vector obtained by decoding the seventh vector through the decoding module, and represents the output of the speech recognition model after decoding the feature vector in the seventh vector (equivalent to the semantic vector representing the second sample speech). The thirteenth vector is a feature vector obtained by decoding the twelfth vector through the decoding module, and represents the output of the speech recognition model after decoding the feature vector in the twelfth vector (equivalent to the semantic vector extracted by the first N layers of convolutional self-attention network representing the second sample speech).

[0128] After determining the seventh vector, the eighth vector, the twelfth vector and the thirteenth vector, the sixth loss value, the seventh loss value and the eighth loss value can be determined; wherein the sixth loss value is obtained by loss calculation based on the seventh vector, the seventh loss value is obtained by loss calculation based on the eighth vector, and the eighth loss value is obtained by loss calculation based on the thirteenth vector.

[0129] In some embodiments, the sixth loss value is obtained by loss calculation based on the seventh vector, and the sixth loss value is used to measure the difference between the seventh vector predicted by the model and the feature vector of the initial sample text, reflecting the performance of the speech recognition model in speech feature extraction. The seventh loss value is obtained by loss calculation based on the eighth vector, and the seventh loss value is used to measure the difference between the eighth vector predicted by the model and the feature vector of the initial sample text, reflecting the performance of the speech recognition model in speech recognition. The eighth loss value is obtained by loss calculation based on the thirteenth vector, and the eighth loss value is used to measure the difference between the thirteenth vector predicted by the model and the feature vector of the initial sample text, reflecting the performance of the initial speech recognition model in the semantic understanding process.

[0130] After calculating the sixth loss value, the seventh loss value, and the eighth loss value, the network parameters in the encoding module and the decoding module can be updated based on the sixth loss value, the seventh loss value, and the eighth loss value to obtain a trained encoding module and a trained decoding module. After that, the first module is integrated between the trained encoding module and the trained decoding module to obtain a speech recognition model, and further training is performed based on the training data using the above steps 101 to 103.

[0131] In some embodiments, the updating process of the network parameters in the encoding module and the decoding module can be implemented by a back propagation algorithm and a gradient descent method, etc., and this process can be iterated until the sixth loss value, the seventh loss value and the eighth loss value all reach a preset value or meet other stop conditions. The speech recognition model when the network parameters in the encoding module and the decoding module meet the second preset training stop condition is determined as the trained speech recognition model, that is, it is determined as a speech recognition model including the trained encoding module and the trained decoding module.

[0132] The second preset training stop condition is a rule that sets the training process to stop in order to prevent overfitting or achieve the expected performance indicators when training the speech recognition model. The second preset training stop condition can be set according to the performance indicators of the speech recognition model (such as loss value, accuracy, etc.). When these performance indicators reach a certain threshold or change trend, the training process will stop. The second preset training stop condition can be that the second comprehensive loss value drops below a specific threshold. The second comprehensive loss value is calculated based on the sixth loss value, the seventh loss value, and the eighth loss value, and can be the weighted sum of the sixth loss value, the seventh loss value, and the eighth loss value.

[0133] The first module is integrated between the trained encoding module and the trained decoding module to obtain the speech recognition model to be trained, which can be the combination of the first module with the trained encoding module and the trained decoding module to form a complete speech recognition model that needs to further use training data to train the first module. In the implementation process, the first module can be inserted between the trained encoding module and the trained decoding module to ensure that the data flow and information transmission are unimpeded between the trained encoding module, the first module and the trained decoding module in turn, forming a continuous model structure, so that the network parameters in the first module can be adjusted through steps 101 to 103 later, so that the first module works in coordination with the trained encoding module and the trained decoding module to improve the overall performance of the speech recognition model.

[0134] In the process of obtaining the speech recognition model through the above steps, not only the loss values ​​(sixth loss value and seventh loss value) are calculated based on the results of the entire encoding and decoding process (seventh vector and eighth vector), but also the loss value (eighth loss value) is additionally calculated based on the output (twelfth vector) of the intermediate layer of the encoding module (the first P layers of convolutional self-attention network) and the decoding result (thirteenth vector); this multi-level loss calculation method helps the encoding module and the decoding module in the speech recognition model to adjust their internal parameters more accurately during the training process, and can utilize the intermediate layer enhancement information of the encoding module to prevent the network parameters from oscillating too much during the training process, and enhance the semantic information of the speech recognition model, and prepare for the subsequent introduction of the first module into the speech recognition model including the encoding module and the decoding module, thereby improving the accuracy and robustness of the speech recognition model at different levels.

[0135] The present application also provides a method for speech recognition. Figure 1G The method includes the following steps 601 to 602:

[0136] Step 601, input the first data into the speech recognition model for third processing to obtain a ninth vector, a tenth vector and an eleventh vector; wherein the speech recognition model includes an encoding module, a first module and a decoding module, the first data includes a first speech and a second text, and the second text includes a fourth word belonging to the second type; the third processing includes: encoding the first speech through the encoding module to obtain a ninth vector; performing feature fusion on the ninth vector and the fourth word through the first module to obtain a tenth vector; decoding the ninth vector and the tenth vector through the decoding module to obtain an eleventh vector.

[0137] The first voice is the voice to be recognized, which may be an audio signal recorded by a user through a microphone or other recording device, and the second type of words are words or phrases that the user expects to receive special attention or recognition during the voice recognition process. The fourth word of the second type may be provided in advance, that is, the second text may be provided in advance.

[0138] The speech recognition model is trained by the speech recognition model training method provided in the embodiment of the present application.

[0139] Step 602: Determine words in the first speech that belong to the second type based on the eleventh vector.

[0140] In some embodiments, after the first speech and the second text are input into the speech recognition model, first, the first speech is encoded using the encoding module in the speech recognition model to obtain a ninth vector; then, the first word module in the speech recognition model is used to perform feature fusion on the ninth vector and the fourth word to obtain a tenth vector; then, the ninth vector and the tenth vector are fused through the decoding module to obtain an eleventh vector; finally, text generation can be performed based on this eleventh vector to determine the words belonging to the second type in the first speech.

[0141] In other embodiments, after fusing the ninth vector and the tenth vector to obtain the eleventh vector, the eleventh vector may be further vector decoded to obtain a decoded vector, and finally text may be generated based on the decoded vector to determine the words belonging to the second type in the first speech.

[0142] Since the speech recognition model trained by the speech recognition model training method provided in the embodiment of the present application can improve the overall performance of speech recognition and enhance the recognition accuracy of the speech recognition model on specific types of words, the words obtained through steps 601 to 602 can highly match the content of the first speech, and can more accurately recognize and convert the words belonging to the second type in the first speech.

[0143] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0144] The embodiment of the present application proposes a speech recognition model training method, which mainly solves the problem of low recognition accuracy of speech recognition models in related technologies, especially the problem of low recognition accuracy of the first word. The first word in the embodiment of the present application specifically refers to a word that frequently appears in the required speech recognition scenario, but rarely appears in the training data, but often appears in specific scenarios.

[0145] First, 50,000 hours of pure speech data are recognized and indexed to obtain the sample text and text index corresponding to each speech (the first sample speech). The first sample speech and sample text are used as the initial training data of the initial speech recognition model (which can be understood as the initial speech recognition module to be trained, which only includes the encoding module and the decoding module).

[0146] Then, the large language model is used to determine the replacement word corresponding to the first word in the sample text: the first word sampling data in the initial training data is predicted using the Chinese open source large language model to obtain synonymous and as homophonic as possible confusion words for replacement. In the implementation process, all the initial training data are first read; each sample text is traversed, and the preset prompt word of the large language model can be "select specific words in this text, including but not limited to domain words, place names, names, etc., which determine the attributes of this text. And for this noun, the top five synonyms or near-synonyms with the highest confidence are given, and they are sorted according to their pronunciation similarity. The number of characters of the synonyms is required to be equal to the number of characters of the original noun". Then all the confusion words obtained by the large language model are stored in a confusion word file, which contains multiple confusion word sets, and the text index of the first sample speech is used as the unique identifier for searching the confusion word set.

[0147] Next, the 50,000 hours of first sample speech and the corresponding sample text are used as initial training data to train the initial speech recognition model. Figure 2A , the training process only uses the encoding module 110 and the decoding module 120, and the encoding module 110 includes a 12-layer convolutional self-attention network (such as Figure 2A The convolutional self-attention network 111-1 ... convolutional self-attention network 111-9 ... convolutional self-attention network 111-12 shown in FIG, the decoding module 120 includes a 6-layer transformer network (such as Figure 2AThe transformer network 121-1... transformer network 121-3... transformer network 121-6 shown in the figure, the encoding module 110 encodes the first sample speech 140 and outputs the acoustic feature vector (seventh vector). After the decoding module 120 predicts and maps (decodes) the acoustic feature vector (seventh vector), it can output the semantic related vector (eighth vector). Based on the label smoothed cross entropy function, the loss of the seventh vector is calculated to obtain the sixth loss value 150-6; based on the attention loss function (label smoothed cross entropy function), the loss of the eighth vector is calculated to obtain the seventh loss value 150-7. In addition, during the training process, the twelfth vector output by the decoding module 120 from the convolutional self-attention network 111-9 is decoded to obtain the thirteenth vector, and the loss of the thirteenth vector is calculated to obtain the intermediate layer attention loss value (eighth loss value 150-8). Finally, the three loss values ​​are summed to obtain a comprehensive loss value. Based on the comprehensive loss value, the initial speech recognition model is trained, and multiple rounds of iterations are performed. When the loss converges, the trained initial speech recognition model is obtained. By comprehensively considering the sixth loss value 150-6, the seventh loss value 150-7, and the eighth loss value 150-8, the intermediate layer enhancement information of the encoding module 110 can be utilized to prevent the network parameters from oscillating too much during the training process, and to enhance the semantic information recognition ability of the initial speech recognition model, and to prepare for the subsequent introduction of the intermediate layer first word network model.

[0148] After that, for the initial training data, traverse each sample text, obtain the above-mentioned confusion word set according to the text index of the sample text, randomly select the confusion word of the sample text with a probability of 3%, and replace the initial first word in the sample text with any confusion word to be selected in the corresponding confusion word set. Traverse the initial training data, randomly select the starting index of the text to be extracted and the number of texts to be extracted in the sample text, and these two values ​​are randomly selected range values; extract text according to these two values, add the extracted text to the training word list (i.e., the first word list), and when the required number of first words is obtained, only the words other than the first words are taken. For example, 3 first words need to be taken, and the preset number of first words is 60. When taking words, it is determined whether 3 first words are taken. If not, continue to traverse. If the 3 first words are obtained, take the remaining first words that are not included from them, and stop traversing when the number of first words is 60. Finally, the training data for training the speech recognition model can be obtained.

[0149] Finally, see Figure 2B, load the trained initial speech recognition model, integrate the first module 130 between the encoding module 110 and the decoding module 120 (i.e., the trained encoding module and the trained decoding module) of the trained initial speech recognition model, train the speech recognition model to be trained, freeze the network parameters of the trained encoding module 110 and the decoding module 120, and only train the network parameters of the first module 130. For the first vector obtained by encoding the first sample speech 140 by the encoding module 110, a loss calculation is performed to obtain a second loss value 150-1; for the second vector obtained by feature fusion of the first vector and the first word 160 by the first module 130, a loss calculation is performed to obtain a third loss value 150-2; for the third vector obtained by decoding the first vector and the second vector by the decoding module 120, a loss calculation is performed to obtain a first loss value 150-3. In addition, the fourth vector and the first word may be feature-fused by the first module to obtain a fifth vector, and a loss calculation may be performed on the fifth vector to obtain a fourth loss value 150-4, wherein the fourth vector is obtained by encoding the first sample speech through the first 9 layers of the convolutional self-attention network (convolutional self-attention network 111-9) of the encoding module 110; the fifth vector is decoded by the decoding module 120 to obtain a sixth vector, and a loss calculation may be performed on the sixth vector to obtain a fifth loss value 150-5. Finally, the speech recognition model is trained based on these five loss values.

[0150] In some embodiments, the first module 130 has a structure as follows: Figure 2C As shown, the first word encoding network 131 in the first module 130 uses a two-layer long short-term memory network (long short-term memory network 170-1 and long short-term memory network 170-2) to perform vector encoding on the first word 160 to obtain a first word encoding vector; the first word encoding vector can be used as a key vector 1322 and a value vector 1323, and the output vector of the encoding module 110 is used as a query vector 1321 and input into the first word fusion network 132, the first word fusion network 132 includes a fusion layer 170, a linear layer 160-1 and a linear layer 160-2; the output of the fusion layer 170 can be output to multiple linear layers (linear layer 160-1 and linear layer 160-2) respectively; the output of the linear layer 160-2 can be decoded by the first word decoding network 133, and the decoding result can be calculated by the connection time series classification algorithm 180 to obtain a third loss value.

[0151] The present application provides a method for speech recognition. Figure 2D, apply the above-mentioned trained speech recognition model, and only load the encoding module 110, the decoding module 120 and part of the first module (the first word encoding network 131 and the first word fusion network 132), but do not load the first word decoding network 133 of the first module 130. The preset weight ratio 134 is set to dynamically adjust the degree of enhancement of the first word during the speech recognition process. When applied in an unknown field, the user sets the first word table of a specific field. The first word table is a text file with only one column of content, which is a series of first words corresponding to this scenario, such as "such and such insurance product". If a voice is received, the 80-dimensional filter bank feature is first extracted, and then the first module 130 is responsible for predicting the first word, and finally the output result is input into the decoding module 120 for processing to predict the corresponding text sequence.

[0152] The present application also provides a robot voice outbound call device 465 for insurance outbound call scenarios. During implementation, a first word list for insurance outbound call scenarios needs to be pre-set, such as "such and such insurance company", "such and such insurance product", etc. The preset first word list is input into the above-mentioned speech recognition model, and then input into the first word encoding network 131 to extract the feature vector of the first word. Figure 2E The schematic diagram of the structure of the robot voice outbound call device shown in the figure, in the actual application of insurance outbound calls, firstly, the voice signal transmitted in real time from the telephone user end is received through the voice acquisition module 4651, and then the voice signal is input into the voice recognition model (i.e., voice recognition module 4652) provided in the embodiment of the present application, and after fast decoding, the output result is input into the intention understanding module 4653, and according to the intention understanding judgment, the judgment logic is matched to the reply to be answered, and the text to be synthesized is generated through the text generation module 4654, and then the voice synthesis module 4655 is used to answer, and a round of reply is completed. This goes back and forth.

[0153] The following is a description of an exemplary structure of the speech recognition model training device 455 provided in the present application as a software module. In some embodiments, Figure 3As shown, the software modules in the speech recognition model training device 455 may include: a first data processing module 4551, which is used to input the training data into the speech recognition model for a first processing to obtain a first vector, a second vector and a third vector; wherein the speech recognition model includes a first module, and the training data includes a first sample speech and a sample text, and the sample text includes a first word belonging to a first type; the first processing includes: encoding the first sample speech to obtain the first vector; performing feature fusion of the first vector and the first word through the first module to obtain the second vector; decoding the first vector and the second vector to obtain the third vector; a loss value determination module 4552, which is used to determine a first loss value based on the third vector; a parameter updating module 4553, which is used to update the parameters of the first module in the speech recognition model based on the first loss value.

[0154] In some embodiments, the first data processing module 4551 is also used to: encode the first word to obtain a feature vector of the first word; perform multi-head attention processing based on the feature vector and the first vector to obtain multiple query vectors, multiple key vectors and multiple value vectors; determine the second vector based on the multiple query vectors, the multiple key vectors and the multiple value vectors.

[0155] In some embodiments, the loss value determination module 4552 is also used to determine the second loss value based on the first vector; determine the third loss value based on the second vector; the parameter updating module 4553 is also used to update the parameters of the first module in the speech recognition model based on the first loss value, the second loss value and the third loss value.

[0156] In some embodiments, the speech recognition model also includes an encoding module and a decoding module, and the first processing also includes: encoding the first sample speech through the encoding module to obtain a fourth vector; performing feature fusion of the fourth vector and the first word through the first module to obtain a fifth vector; decoding the fifth vector through the decoding module to obtain a sixth vector; the first data processing module 4551 is also used to: determine a fourth loss value based on the fifth vector; and determine a fifth loss value based on the sixth vector; the updating of the parameters of the first module in the first speech recognition model based on the first loss value, the second loss value and the third loss value includes: updating the parameters of the first module in the speech recognition model based on the first loss value, the second loss value, the third loss value, the fourth loss value and the fifth loss value.

[0157] In some embodiments, the first data processing module 4551 is also used to: before inputting the training data into the speech recognition model for the first processing, inputting the second sample speech into the speech recognition model for the second processing to obtain the seventh vector and the eighth vector; wherein the second processing includes: encoding the second sample speech through the encoding module to obtain the seventh vector; decoding the seventh vector through the decoding module to obtain the eighth vector; determining the sixth loss value based on the seventh vector; determining the seventh loss value based on the eighth vector; and updating the parameters of the encoding module and the decoding module in the speech recognition model based on the sixth loss value and the seventh loss value.

[0158] In some embodiments, the device also includes: a sample text generation module, which is used to perform speech recognition on the first sample speech to obtain a first text; based on a preset prompt word, extract a second word belonging to the first type from the first text; replace a preset number of second words in the first text with a third word to obtain the sample text; wherein the third word is a word with the same number of words and a similar pronunciation as the second word; the first word in the sample text includes the third word and the second word that has not been replaced.

[0159] In some embodiments, the first data processing module 4551 is also used to: process the first vector using a first weight to obtain a first weighted vector, and process the second vector using a second weight to obtain a second weighted vector; wherein there is a preset weight ratio between the first weight and the second weight; and determine the third vector based on the first weighted vector and the second weighted vector.

[0160] The following is a description of an exemplary structure of the speech recognition device 445 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 4As shown, the software modules in the speech recognition device 445 may include: The embodiment of the present application further provides a speech recognition device, including: a second data processing module 4451, used to input the first data into the speech recognition model for third processing to obtain an eighth vector, a ninth vector and a tenth vector; wherein the speech recognition model includes an encoding module, a first module and a decoding module, the first data includes a first speech and a second text, and the second text includes a fourth word belonging to the second type; the third processing includes: encoding the first speech by the encoding module to obtain the eighth vector; performing feature fusion on the eighth vector and the fourth word by the first module to obtain the ninth vector; decoding the eighth vector and the ninth vector by the decoding module to obtain the tenth vector; a determination module 4452, used to determine the words belonging to the second type in the first speech based on the tenth vector.

[0161] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0162] Correspondingly, an embodiment of the present application provides an electronic device, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the electronic device 1200 includes at least: a processor 1201 , a communication interface 1202 , and a storage medium 1203 configured to store executable instructions, wherein: the processor 1201 generally controls the overall operation of the electronic device 1200 .

[0163] The communication interface 1202 enables the electronic device to communicate with other terminals or servers through a network.

[0164] The storage medium 1203 is configured to store instructions and applications executable by the processor 1201, and can also cache data to be processed or processed by the processor 1201 and each module in the electronic device 1200, which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0165] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the speech recognition model training method and the speech recognition method described in the embodiment of the present application.

[0166] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the speech recognition model training method and speech recognition method provided in the embodiment of the present application.

[0167] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0168] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0169] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0170] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0171] In summary, the training data is processed by the encoding module, the first module and the decoding module to obtain multiple vectors, and the network parameters in the first module are updated based on the loss values ​​of these vectors. Since the second loss value in these loss values ​​considers the first sample speech, the third loss value considers the first sample speech and the first word in the sample text, and the first loss value considers the first sample speech and the sample text at the same time, the parameters of the first module are updated based on these loss values, so that the trained speech recognition model can more comprehensively consider the association between speech and text. In addition, since the second vector is obtained by fusing the speech features (i.e., the first vector) of the first sample speech with the first word, the third loss value calculated based on the second vector can evaluate the effect of the fusion of the speech features with the word features of the first word, and after updating the parameters of the first module based on the third loss value, the trained speech recognition model can more accurately fuse the speech features and the word features, thereby improving the recognition accuracy of the first word. It can be seen that the embodiment of the present application enables the trained speech recognition model to learn more abundant information through the joint optimization of multiple loss values, thereby improving the overall recognition performance of the trained speech recognition model.

[0172] In the embodiments of the present application, related data such as the user's voice to be recognized is involved. When the embodiments of the present application are applied to specific products or technologies, the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0173] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A speech recognition model training method, characterized in that: include: Input the training data into the speech recognition model for first processing to obtain a first vector, a second vector and a third vector; wherein the speech recognition model includes a first module, the training data includes a first sample speech and a sample text, and the sample text includes a first word belonging to a first type; the first processing includes: encoding the first sample speech to obtain the first vector; performing feature fusion on the first vector and the first word through the first module to obtain the second vector; decoding the first vector and the second vector to obtain the third vector; determining a first loss value based on the third vector; Based on the first loss value, the parameters of the first module in the speech recognition model are updated.

2. The method according to claim 1, characterized in that The step of fusing the first vector and the first word to obtain the second vector includes: Encoding the first word to obtain a feature vector of the first word; Perform multi-head attention processing based on the feature vector and the first vector to obtain multiple query vectors, multiple key vectors, and multiple value vectors; The second vector is determined based on the plurality of query vectors, the plurality of key vectors, and the plurality of value vectors.

3. The method according to claim 1, characterized in that The method further comprises: Determine a second loss value based on the first vector; determine a third loss value based on the second vector; The updating of the parameters of the first module in the speech recognition model based on the first loss value includes: Based on the first loss value, the second loss value and the third loss value, the parameters of the first module in the speech recognition model are updated.

4. The method according to claim 3, characterized in that The speech recognition model further includes an encoding module and a decoding module, and the first processing further includes: encoding the first sample speech by the encoding module to obtain a fourth vector; fusing the fourth vector and the first word by the first module to obtain a fifth vector; decoding the fifth vector by the decoding module to obtain a sixth vector; The method further includes: determining a fourth loss value based on the fifth vector; and determining a fifth loss value based on the sixth vector; The updating of the parameters of the first module in the first speech recognition model based on the first loss value, the second loss value and the third loss value includes: updating the parameters of the first module in the speech recognition model based on the first loss value, the second loss value, the third loss value, the fourth loss value and the fifth loss value.

5. The method according to claim 4, characterized in that Before inputting the training data into the speech recognition model for first processing, the method further includes: Inputting the second sample speech into the speech recognition model for second processing to obtain a seventh vector and an eighth vector; wherein the second processing includes: encoding the second sample speech by the encoding module to obtain the seventh vector; decoding the seventh vector by the decoding module to obtain the eighth vector; Determine a sixth loss value based on the seventh vector; determine a seventh loss value based on the eighth vector; Based on the sixth loss value and the seventh loss value, the parameters of the encoding module and the decoding module in the speech recognition model are updated.

6. The method according to claim 1, characterized in that The method further comprises: Performing speech recognition on the first sample speech to obtain a first text; Based on a preset prompt word, extracting a second word belonging to the first type from the first text; Replacing a preset number of second words in the first text with third words to obtain the sample text; The third word is a word with the same number of characters and similar pronunciation as the second word; the first word in the sample text includes the third word and the second word that has not been replaced.

7. The method according to any one of claims 1 to 6, characterized in that: The decoding of the first vector and the second vector to obtain the third vector includes: The first vector is processed with a first weight to obtain a first weighted vector, and the second vector is processed with a second weight to obtain a second weighted vector; wherein a preset weight ratio exists between the first weight and the second weight; The third vector is determined based on the first weighting vector and the second weighting vector.

8. A speech recognition method, characterized in that: The method comprises: Inputting the first data into the speech recognition model for third processing to obtain a ninth vector, a tenth vector and an eleventh vector; wherein the speech recognition model includes an encoding module, a first module and a decoding module, the first data includes a first speech and a second text, and the second text includes a fourth word belonging to the second type; the third processing includes: encoding the first speech by the encoding module to obtain the ninth vector; performing feature fusion on the ninth vector and the fourth word by the first module to obtain the tenth vector; decoding the ninth vector and the tenth vector by the decoding module to obtain the eleventh vector; Based on the eleventh vector, words in the first speech belonging to the second type are determined.

9. A speech recognition model training device, characterized in that: The device comprises: A first data processing module is used to input the training data into the speech recognition model for first processing to obtain a first vector, a second vector and a third vector; wherein the speech recognition model includes the first module, the training data includes a first sample speech and a sample text, and the sample text includes a first word belonging to a first type; the first processing includes: encoding the first sample speech to obtain the first vector; performing feature fusion on the first vector and the first word through the first module to obtain the second vector; decoding the first vector and the second vector to obtain the third vector; A loss value determining module, configured to determine a first loss value based on the third vector; A parameter updating module is used to update the parameters of the first module in the speech recognition model based on the first loss value.

10. A speech recognition device, characterized in that: The device comprises: a second data processing module, configured to input the first data into a speech recognition model for third processing to obtain a ninth vector, a tenth vector, and an eleventh vector; wherein the speech recognition model comprises an encoding module, a first module, and a decoding module, the first data comprises a first speech and a second text, and the second text comprises a fourth word belonging to a second type; the third processing comprises: encoding the first speech by the encoding module to obtain the ninth vector; performing feature fusion on the ninth vector and the fourth word by the first module to obtain the tenth vector; and decoding the ninth vector and the tenth vector by the decoding module to obtain the eleventh vector; A determination module is used to determine the words in the first speech that belong to the second type based on the eleventh vector.

11. An electronic device, characterized in that: include: A memory for storing computer executable instructions; The processor is used to implement the speech recognition model training method described in any one of claims 1 to 8, or implement the speech recognition method described in claim 9 when executing the computer executable instructions stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the speech recognition model training method described in any one of claims 1 to 8 is implemented, or the speech recognition method described in claim 9 is implemented.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the speech recognition model training method described in any one of claims 1 to 8 is implemented, or the speech recognition method described in claim 9 is implemented.

Images