Model training method, speech recognition method, and related devices

Abstract

The present disclosure provides a model training method, a speech recognition method and related devices, which are related to the technical field of data processing, and in particular to the technical fields of artificial intelligence, computer vision, speech technology, intelligent search and the like. The specific implementation scheme is as follows: inputting a mouth shape sample sequence into a mouth shape processing model to obtain a first dictionary code prediction result predicted based on the mouth shape sample sequence; determining a loss value based on the first dictionary code prediction result and a second dictionary code prediction result; the second dictionary code prediction result is determined based on a target text corresponding to the mouth shape sample sequence; adjusting model parameters of the mouth shape processing model based on the loss value to obtain a mouth shape interpretation model; wherein the mouth shape interpretation model is used to assist speech recognition. The mouth shape interpretation model trained by the method can adapt to any speech recognition network and can realize plug and play.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to the technical field of artificial intelligence, computer vision, speech technology, intelligent search, etc. BACKGROUND

[0002] Speech recognition is a key technology in the field of artificial intelligence, which enables machines to understand and process human speech, converting it into text or executing specific commands. This technology spans multiple disciplines such as signal processing, pattern recognition, probability theory, information theory, and linguistics.

[0003] In recent years, with the breakthrough progress of deep learning technology, this technology has been applied in many industries, including but not limited to industrial automation, household appliances, automobiles, electronic consumer goods, etc. SUMMARY

[0004] The present disclosure provides a model training method, a speech recognition method and related devices.

[0005] According to an aspect of the present disclosure, a model training method is provided, comprising:

[0006] inputting a mouth shape sample sequence into a mouth shape processing model to obtain a first dictionary code prediction result predicted based on the mouth shape sample sequence;

[0007] determining a loss value based on the first dictionary code prediction result and a second dictionary code prediction result; the second dictionary code prediction result is determined based on a target text corresponding to the mouth shape sample sequence;

[0008] adjusting model parameters of the mouth shape processing model based on the loss value to obtain a mouth shape interpretation model; wherein the mouth shape interpretation model is used to assist in speech recognition.

[0009] According to another aspect of the present disclosure, a speech recognition method is provided, which is applied to a mouth shape interpretation model trained by the foregoing method, comprising:

[0010] obtaining a target speech of a target object and a mouth shape image sequence corresponding to the target speech;

[0011] processing the mouth shape image sequence based on the mouth shape interpretation model to obtain a third dictionary code prediction result; and

[0012] inputting the target speech into a target speech recognition network to obtain a fourth dictionary code prediction result;

[0013] fusing the third dictionary code prediction result and the fourth dictionary code prediction result to obtain a fused code prediction result;

[0014] The fusion coding prediction result is parsed based on a dictionary to obtain text information corresponding to the target speech.

[0015] According to another aspect of the present disclosure, a model training apparatus is provided, comprising:

[0016] An input module is configured to input a lip shape sample sequence to a lip shape processing model to obtain a first dictionary coding prediction result predicted based on the lip shape sample sequence;

[0017] A determination module is configured to determine a loss value based on the first dictionary coding prediction result and a second dictionary coding prediction result, the second dictionary coding prediction result being determined based on target text corresponding to the lip shape sample sequence;

[0018] An optimization module is configured to adjust model parameters of the lip shape processing model based on the loss value to obtain a lip shape interpretation model, wherein the lip shape interpretation model is configured to assist speech recognition.

[0019] According to another aspect of the present disclosure, a speech recognition apparatus is provided, comprising a lip shape interpretation model trained by the aforementioned model training apparatus, comprising:

[0020] An acquisition module is configured to acquire target speech of a target object and a lip shape image sequence corresponding to the target speech;

[0021] A first prediction module is configured to process the lip shape image sequence based on the lip shape interpretation model to obtain a third dictionary coding prediction result; and

[0022] A second prediction module is configured to input the target speech into a target speech recognition network to obtain a fourth dictionary coding prediction result;

[0023] A fusion module is configured to fuse the third dictionary coding prediction result and the fourth dictionary coding prediction result to obtain a fusion coding prediction result;

[0024] An identification module is configured to parse the fusion coding prediction result based on a dictionary to obtain text information corresponding to the target speech.

[0025] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0026] at least one processor; and

[0027] a memory in communication with the at least one processor; wherein

[0028] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the embodiments of the present disclosure.

[0029] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method according to any of the embodiments of the present disclosure.

[0030] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any of the embodiments of the present disclosure.

[0031] According to another aspect of the present disclosure, there is provided a vehicle comprising the electronic device as described above.

[0032] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0034] Figure 1 is a schematic diagram of speech recognition combined with a mouth shape according to an embodiment of the present disclosure;

[0035] Figure 2 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0036] Figure 3 is a framework diagram of a trained model according to an embodiment of the present disclosure;

[0037] Figure 4 is a flowchart of determining a reference loss according to an embodiment of the present disclosure;

[0038] Figure 5 is a flowchart of a speech recognition method according to an embodiment of the present disclosure;

[0039] Figure 6 is a framework diagram of speech recognition according to an embodiment of the present disclosure;

[0040] Figure 7 is a structural diagram of a mouth shape interpretation model according to an embodiment of the present disclosure;

[0041] Figure 8 is a structural diagram of a model training device according to an embodiment of the present disclosure;

[0042] Figure 9 is a structural diagram of a speech recognition device according to an embodiment of the present disclosure;

[0043] Figure 10 is a block diagram of an electronic device for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0044] Exemplary embodiments of the present disclosure are described herein below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding them. These should be considered in a descriptive sense only and not limiting. As such, various changes and modifications can be made to the embodiments described herein by those of ordinary skill in the art without departing from the scope of the present disclosure. Also, for the purpose of clarity and the brevity of description, descriptions of well-known functions and constructions are omitted from the following description.

[0045] The terms "first", "second", and the like in the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a particular order or sequence. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, inclusion of a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0046] The breakthrough of deep learning technology greatly improves the accuracy and efficiency of speech recognition, but in some complex scenarios, the accuracy of the current speech recognition result needs to be improved.

[0047] For example, in-car voice has become an important configuration in the automotive market under the trend of intelligentization. Drivers can control the navigation system, adjust music and temperature, etc. through voice instructions without distracting attention, improving driving safety. However, the related technical solutions are limited by problems such as high noise in the car, difficulty in distinguishing homophonic characters, loss of context information, etc., and are prone to voice misrecognition. Therefore, the accuracy of speech recognition needs to be improved.

[0048] In view of this, the embodiments of the present disclosure provide a scheme of mouth shape prediction combined with speech recognition, which can further improve the accuracy of speech recognition with the help of mouth shape prediction.

[0049] The idea is as shown in Figure 1 The mouth shape image sequence of the speech is collected at the same time as the speech is collected. A mouth shape interpretation model can be provided to analyze and infer the mouth shape image sequence to obtain a first speech recognition result based on mouth shape prediction. At the same time, the collected speech is input into an ASR (Automatic Speech Recognition) network to obtain a second speech recognition result, and then the first speech recognition result and the second speech recognition result can be fused to obtain a final speech recognition result.

[0050] In order to accurately use the mouth shape prediction to assist in improving the accuracy of speech recognition, the model training method provided by the embodiment of the present disclosure is used to train the aforementioned mouth shape interpretation model. As shown in Figure 2 The flowchart of the model training method provided by the embodiment of the present disclosure includes the following contents:

[0051] S201, inputting a mouth shape sample sequence into a mouth shape processing model to obtain a first dictionary code prediction result predicted based on the mouth shape sample sequence.

[0052] The mouth shape sample sequence is an image sequence constructed by multiple lip images. The multiple lip images conform to the pronunciation rules of natural speech. The face image of the target object pronouncing the speech can be synchronously collected when the speech is collected to obtain the multiple lip images, so as to construct the mouth shape sample sequence.

[0053] In the field of speech recognition, a dictionary is an important component, which usually refers to a mapping relationship that maps pronunciation (usually phonemes or letters) to a corresponding word or character sequence. The dictionary plays a bridge role in the speech recognition system, connecting the acoustic model and the language model.

[0054] The first dictionary code prediction result is the probability of mapping the mouth shape into a candidate word or character sequence. That is, the mouth shape processing model is a regression model, which is used to map the mouth shape sample sequence to the corresponding word or character sequence in the dictionary.

[0055] S202, determining a loss value based on the first dictionary code prediction result and a second dictionary code prediction result; the second dictionary code prediction result is determined based on the target text corresponding to the mouth shape sample sequence.

[0056] The second dictionary code prediction result is the index sequence of the Tokenizer dictionary, that is, the probability of mapping the target text into a candidate word or character sequence. Unlike the first dictionary code prediction result, the first dictionary code prediction result is obtained by predicting the mouth shape, and the second dictionary code prediction result is obtained based on the target text. The target text is the text expressed by the mouth shape sample sequence.

[0057] S203, adjusting the model parameters of the mouth shape processing model based on the loss value to obtain a mouth shape interpretation model; wherein the mouth shape interpretation model is used to assist in speech recognition.

[0058] In the embodiment of the present disclosure, the first dictionary code prediction result obtained based on the mouth shape and the second dictionary code prediction result obtained based on the target text are used for model training, so that the mouth shape processing model can perform comparative learning based on the target text, so that the mouth shape interpretation result approaches the true value, thereby improving the accuracy of the mouth shape interpretation model in speech recognition.

[0059] After the mouth shape interpretation model is trained, the speech can be understood through the mouth shape during speech recognition, thereby providing a multi-modal speech recognition manner, which can improve the accuracy of speech recognition compared with single-modal speech recognition. It is especially suitable for scenes with high noise and multi-sound characters, can better interact with users, and improve user experience.

[0060] In addition, in the training model stage in the embodiment of the present disclosure, the target text is combined with the mouth shape sample sequence for training, and the entire training process can not involve the speech recognition model (i.e., ASR), which enables the mouth shape interpretation model trained by the present disclosure to adapt to any ASR, avoids the cumbersome operation of retraining the mouth shape interpretation model for a new ASR, and can save time cost and computer resources.

[0061] In some embodiments, as shown in FIG. 1, the mouth shape processing model includes a mouth shape encoder, a mouth shape decoder, and a word classifier connected after the mouth shape decoder, wherein Figure 3

[0062] The mouth shape encoder is configured to process the mouth shape sample sequence to obtain mouth shape features.

[0063] The mouth shape decoder is configured to process the mouth shape features to obtain intermediate features.

[0064] The word classifier is configured to process the intermediate features to predict a first dictionary code prediction result.

[0065] The mouth shape encoder and the mouth shape decoder are both regression networks. The mouth shape decoder is a tokenizer-level regression network, so as to identify the intermediate features for the word classifier from the mouth shape sample sequence.

[0066] In the embodiment of the present disclosure, the mouth shape decoder can better solve the homonym problem by combining the mouth shape context. The mouth shape encoder and the mouth shape decoder can convert the mouth shape sample sequence to the dictionary space, so as to facilitate the classifier to predict the first dictionary code prediction result. In addition, the mouth shape decoder and the mouth shape encoder are jointly trained, which can also extract important features from the mouth shape sample sequence to generate the mouth shape features, so as to improve the mouth shape prediction accuracy of the mouth shape interpretation model.

[0067] In some embodiments, in order to improve the training efficiency of the model and improve the accuracy of mouth shape prediction, the present embodiment not only uses the prediction loss between the first dictionary code prediction result and the second dictionary code prediction result to adjust the model parameters, but also introduces a reference loss to assist in training the mouth shape processing model from multiple angles or dimensions. In implementation, the reference loss can include at least one of the following:

[0068] ​1) a first loss between the lip feature of the lip sample sequence and the text feature of the target text;

[0069] Thus, the lip feature and the text feature are mapped to the same feature space by the first loss, or the lip feature is aligned with the text feature, which makes the lip sample sequence and the target text aligned in the feature dimension, and helps to improve the rationality of the lip feature extraction, so that the lip processing model learns how to extract reliable and important feature information from the lip sample sequence for lip-based speech recognition.

[0070] In implementation, as shown in the following: Figure 4

[0071] S401, obtaining the lip feature obtained by processing the lip sample sequence by the lip encoder in the lip processing model.

[0072] S402, obtaining the text feature by processing the target text based on the text feature extractor;

[0073] S403, determining the first loss between the lip feature and the text feature.

[0074] Among them, the MSE (Mean Squared Error, Mean Squared Error) regression loss between the lip feature and the text feature can be determined as the first loss, and the L1loss (Mean Absolute Error, MAE, Mean Absolute Error loss) / huber loss (huber loss) between the lip feature and the text feature can also be determined as the first loss.

[0075] As shown in the following: Figure 3 The target text extracts the text feature through the text feature extractor. In implementation, the text feature extractor can be a (BERT (Bidirectional Encoder Representations from Transformers, Bidirectional Encoder Representations from Transformers)) model, or a large language model. The large language model refers to a natural language processing model with a large number of parameters trained by deep learning technology. Because of its large size and complex architecture, these models can capture and understand the subtle differences of natural language, and thus perform well in various language tasks.

[0076] Comparing the text feature and the lip feature is beneficial to extract reasonable lip features under the guidance of the target text to improve the model training efficiency, so as to improve the accuracy of lip-based speech recognition.

[0077] 2) a second loss between the phoneme prediction result of the lip sample sequence and the phoneme index sequence of the target text.

[0078] ​A phoneme is a basic concept in phonetics, which is an abstract concept of the smallest unit of sound in human language. A phoneme is the smallest unit of speech that can distinguish the meaning of a word. Different phonemes can distinguish the meaning of a word. In speech recognition technology, phonemes are usually used as modeling units of acoustic models to help the system recognize and understand speech signals.

[0079] Thus, by the second loss defined by the phonemes, the mouth shape processing model can efficiently and accurately assist in speech recognition, and the performance and practicality of speech recognition can be improved.

[0080] In implementation, as shown in the following formula, the second loss can be determined based on the following method, including: Figure 4

[0081] S404, input the mouth shape feature into the phoneme recognition network for recognizing phonemes to obtain the phoneme prediction result of the mouth shape sample sequence.

[0082] The phoneme recognition network is used to convert the mouth shape feature to the phoneme space to obtain the probability prediction of the phoneme sequence as the phoneme prediction result.

[0083] S405, compare the phoneme prediction result with the phoneme index sequence of the target text to obtain the second loss.

[0084] In implementation, the cross-entropy loss between the phoneme prediction result and the phoneme index sequence can be determined to obtain the second loss.

[0085] In the embodiments of the present disclosure, the mouth shape feature is converted to the phoneme space based on the phoneme recognition network, so as to be compared with the phoneme index sequence of the target text, thereby improving the training efficiency of the mouth shape processing model and improving the accuracy of language recognition based on the mouth shape of the mouth shape processing model.

[0086] In implementation, as shown in the following formula, the phoneme recognition network includes a phoneme-level regression network and a phoneme classifier. Figure 3

[0087] The phoneme-level regression network is used to process the mouth shape feature to obtain the phoneme feature.

[0088] The phoneme classifier is used to process the phoneme feature to predict the phoneme prediction result of the output mouth shape sample sequence.

[0089] As shown in the following formula, the target file is converted into a phoneme sequence, and thus the phoneme index sequence of the target text can be obtained. Figure 3

[0090] The phoneme-level regression network can convert the mouth shape feature to the phoneme feature space, so as to align the mouth shape and the target text from the phoneme perspective, thereby improving the speech recognition capability of the mouth shape processing model.

[0091] ​​​The preceding text explained that the first loss and / or the second loss can be used as reference losses to assist in training the lip-sync processing model. In practice, after determining the reference loss, the prediction loss between the first dictionary encoding prediction result and the second dictionary encoding prediction result, along with the reference loss, can be weighted and summed to obtain the loss value.

[0092] The weights can be set based on empirical values. In practice, to improve model accuracy, the weights of the reference loss and prediction loss are obtained by training the lip-sync processing model. That is, when adjusting the model parameters of the lip-sync processing network based on the loss values, the weights can be adjusted simultaneously as model parameters. For example, the reference loss and prediction loss can be input into a fully connected layer to adjust the contribution of different losses to the loss value through the weights within the fully connected layer.

[0093] Therefore, by learning the weights of different losses, we can reasonably adjust the impact of different losses on the model parameters, so that the intermediate and final results of the lip-sync processing model can approach the target text, thereby improving the training efficiency of the model.

[0094] When using the first loss and the second loss as reference losses, the first loss and the second loss have their own weights, and the weights of each loss can also be obtained based on training.

[0095] In summary, as Figure 3 As shown in this embodiment, three loss functions are introduced to train the lip-sync processing model. During the training phase, Figure 3 The model parameters of each neural network involved can be adjusted to optimize the lip-reading processing model. For example, by comparing lip-reading features and text features, the lip-reading encoder can learn how to transform lip-reading sample sequences into a suitable lip-reading feature space under the guidance of text features. Based on the loss between phonemes, the lip-reading encoder can align the phonemes with the target text. Finally, guided by these two losses, the lip-reading encoder learns how to extract important lip-reading features from the lip-reading sample sequence. In addition to comparing and learning intermediate results, in this embodiment, the lip-reading decoder and word classifier of the lip-reading processing model are optimized through token-level prediction loss, making the first dictionary encoding prediction result based on lip-reading closer to the second dictionary encoding prediction result of the target text. Throughout the entire training process, the lip-reading processing model is comprehensively constrained and learned from the intermediate levels to the final result, thereby improving the recognition accuracy and training efficiency of the lip-reading processing model and obtaining the final desired lip-reading interpretation model.

[0096] Furthermore, in this embodiment of the disclosure, the entire training process does not require adjustment of the speech recognition network (i.e., ASR), and the resulting lip-reading model can be quickly adapted to any speech recognition network, enabling plug-and-play functionality to help improve speech recognition accuracy.

[0097] Since the training process relies on the dictionary, as society progresses and develops, new words will continue to emerge, and in the case of having new words added to the dictionary, the lip-reading model is continuously optimized based on the model parameters of the lip-reading model to adapt the lip-reading model to the new words.

[0098] That is, in the case of having new words added, the model parameters previously trained by the lip-reading model still have important value, and the lip-reading model can be iteratively trained based on the original parameters of the lip-reading model to obtain a lip-reading model that adapts to new words. Thus, the lip-reading model provided in the embodiments of the present disclosure can be iteratively optimized as the dictionary changes, without the need for retraining.

[0099] Taking a vehicle terminal as an example, due to the problems of large in-vehicle noise, difficulty in distinguishing homophonic characters, and loss of context information, it is difficult to use a single model to predict accurate speech-to-text conversion, which will to some extent lead to inaccurate speech recognition.

[0100] Based on the lip-reading model provided in the embodiments of the present disclosure, in the application of in-vehicle speech recognition, the speaker's face and lip movements can be captured by the in-vehicle camera, and the lip-reading model in the embodiments of the present disclosure can be combined with the ASR network to improve the accuracy of speech recognition in the in-vehicle environment. For example, in a noisy vehicle environment, some speech cannot be recognized due to the influence of noise, and the lip-reading model can be used to analyze the lost speech to improve the accuracy of speech recognition. For another example, the lip-reading model processes a sequence of lip movement images, which contains context information and can effectively distinguish homophonic characters to assist in improving the accuracy of speech recognition.

[0101] Thus, based on the same technical concept, the embodiments of the present disclosure also provide a speech recognition method, as shown in Figure 5 The method comprises the following contents:

[0102] S501, obtaining target speech of a target object and a sequence of lip movement images corresponding to the target speech.

[0103] S502, processing the sequence of lip movement images based on a lip-reading model to obtain a third dictionary code prediction result.

[0104] S503, inputting the target speech into a target speech recognition network to obtain a fourth dictionary code prediction result.

[0105] In the embodiments of the present disclosure, the execution time of S502 and S503 is not limited.

[0106] S504, fusing the third dictionary code prediction result and the fourth dictionary code prediction result to obtain a fused code prediction result.

[0107] S505, parse the fusion coding prediction result based on the dictionary to obtain text information corresponding to the target speech.

[0108] The text information can be used for subsequent execution of voice instructions, or execution of other text information-based processing, for example, further analysis and processing of the text information can be performed using a large language model. The specific use of the text information is not limited in the embodiments of the present disclosure.

[0109] In the embodiments of the present disclosure, the mouth shape image sequence covers the context information content based on the mouth shape. A speech recognition result is parsed through a mouth shape interpretation model, and another speech recognition result is parsed by combining the information of the speech mode. The two speech recognition results are comprehensively processed to obtain the final speech recognition result. The whole process is complementary, which can solve the homonym problem and solve the problem of missing context in the audio caused by the lack of speech in a noisy environment, thereby improving the accuracy of speech recognition.

[0110] The third dictionary coding prediction result is a probability, which represents the probability value that the word sequence expressed in the mouth shape image sequence belongs to the corresponding word sequence in the dictionary. Similarly, the fourth dictionary coding prediction result represents the probability value that the word sequence expressed in the speech belongs to the corresponding word sequence in the dictionary.

[0111] In implementation, the third dictionary coding prediction result and the fourth dictionary coding prediction result are fused to obtain the fusion coding prediction result, which can be implemented by performing a point multiplication operation on the same bits in the third dictionary coding prediction result and the fourth dictionary coding prediction result to obtain the fusion coding prediction result.

[0112] In implementation, the fusion coding prediction result obtained by the point multiplication operation can be subjected to an argmax operation to obtain the predicted Tokenizer coding sequence index, and then the Tokenizer coding dictionary is looked up to obtain the final ASR recognition result (i.e., the text information). The argmax is a commonly used operation in mathematics and computer science, which is used to find the index or position of the maximum value from a set of numerical values.

[0113] In the embodiments of the present disclosure, the results based on the mouth shape branch recognition and the results based on the speech branch recognition can be fused through the point multiplication operation to improve the accuracy of speech recognition.

[0114] In other embodiments, the third dictionary coding prediction result and the fourth dictionary coding prediction result are fused to obtain the fusion coding prediction result, which can also be implemented as:

[0115] In step A1, the third dictionary coding prediction result and the fourth dictionary coding prediction result are subjected to an accumulation operation on the same bits to obtain an intermediate coding result.

[0116] Wherein, the intermediate coding result is obtained by accumulation, and then the intermediate coding result is input into a softmax network to obtain the intermediate coding result.

[0117] Step A2, converting the intermediate coding result into a probability distribution to obtain a fusion coding prediction result.

[0118] Wherein, argmax operation can be performed on the fusion coding prediction result to obtain a predicted Tokenizer coding sequence index, and then the Tokenizer coding dictionary is looked up to obtain the final ASR recognition result (i.e., text information).

[0119] In the embodiments of the present disclosure, through accumulation and conversion operations, the results based on the lip shape branch recognition and the results based on the speech branch recognition can be fused to improve the accuracy of speech recognition.

[0120] In some embodiments, the target speech of the target object and the lip shape image sequence corresponding to the target speech can be implemented as:

[0121] Step B1, obtaining a video of the target object acquired synchronously with the target speech.

[0122] For example, the face image of the speaker can be collected by the camera in the vehicle to obtain the video of the target object. Since the processing result of the speech recognition network score needs to be adapted, in step B2, the video is frame-extracted based on the target sampling rate to obtain the lip shape image sequence.

[0123] Wherein, the target sampling rate = the length of the speech sequence / the down-sampling rate of the speech feature extraction in the speech recognition network. The down-sampling rate of the speech feature extraction in the speech recognition network is a fixed value of the speech recognition network.

[0124] In the embodiments of the present disclosure, by down-sampling the video based on the target sampling rate, the extracted lip shape image sequence can be adapted to the characteristics of the speech processing of the speech recognition network, so that the lip shape image sequence can be aligned with the processing result of the speech recognition network, and the accuracy of speech recognition is improved.

[0125] In some embodiments, the third dictionary coding prediction result can be obtained by processing the lip shape image sequence based on a lip shape interpretation model.

[0126] Step C1, extracting an external lip shape feature point from each frame of image in the lip shape image sequence.

[0127] In implementation, the external lip shape feature point can be extracted from each frame of image in the lip shape image sequence, so as to extract the external contour feature of the lip shape.

[0128] Step C2, normalizing the distance between the lip shape feature points to obtain a mouth opening coefficient sequence.

[0129] In implementation, because the mouth shape feature points include multiple points, the distance between each two points in the multiple points can be calculated, and then the maximum distance is used for normalization processing to obtain the mouth opening coefficient sequence.

[0130] Step C3, inputting the mouth opening coefficient sequence into the mouth shape interpretation model to obtain a third dictionary code prediction result.

[0131] In the embodiments of the present disclosure, the mouth shape image sequence is initialized to reduce the data volume of the data processed by the mouth shape interpretation model, so as to improve the processing efficiency of the mouth shape interpretation model and improve the accuracy of speech recognition.

[0132] Taking the vehicle-mounted scene as an example, the speech recognition process provided by the embodiments of the present disclosure is as shown in Figure 6 .

[0133] a. The face information of the target object is captured by the vehicle-mounted camera, and the frame extraction is performed on the face image sequence in combination with the feature extraction result of the speech end ASR network (frame number = speech sequence length / ASR network speech feature extraction down-sampling ratio), and then the mouth shape key point recognition processing is performed on each frame of face image, so as to obtain the mouth shape feature points, and then the distance between the mouth shape feature points corresponding to the upper and lower lips is normalized to the mouth opening coefficient sequence between 0 and 1;

[0134] b. The mouth opening coefficient sequence obtained above is input into the aforementioned mouth shape interpretation network (the structure of which is as shown in Figure 7 ) trained to obtain the Tokenizer code prediction result 1 (probability sequence) of the mouth shape branch;

[0135] c. The speech of the speaker is preprocessed (such as frame division, noise reduction, etc.) and input into the ASR network to obtain the candidate Tokenizer code prediction result 2 (probability sequence) of the speech end;

[0136] d. The Tokenizer code prediction result 1 and the candidate Tokenizer code prediction result 2 are multiplied point by point to obtain the final probability prediction sequence, and then the probability prediction sequence is subjected to argmax operation to obtain the predicted Tokenizer code sequence index, and then the Tokenizer code dictionary is looked up to obtain the final ASR recognition result.

[0137] Based on the same technical concept, the embodiments of the present disclosure also provide a model training device 800, as shown in Figure 8 , comprising:

[0138] An input module 801 is configured to input a mouth shape sample sequence into a mouth shape processing model to obtain a first dictionary code prediction result predicted based on the mouth shape sample sequence;

[0139] The determining module 802 is configured to determine a loss value based on the first dictionary encoding prediction result and a second dictionary encoding prediction result, where the second dictionary encoding prediction result is determined based on target text corresponding to the lip sample sequence.

[0140] The optimization module 803 is configured to adjust model parameters of the lip processing model based on the loss value to obtain a lip interpretation model, where the lip interpretation model is used to assist speech recognition.

[0141] In some embodiments, the lip processing model comprises a lip encoder, a lip decoder, and a word classifier connected after the lip decoder.

[0142] The lip encoder is configured to process the lip sample sequence to obtain lip features.

[0143] The lip decoder is configured to process the lip features to obtain intermediate features.

[0144] The word classifier is configured to process the intermediate features to predict the first dictionary encoding prediction result.

[0145] In some embodiments, the determining module comprises:

[0146] An auxiliary unit configured to determine a reference loss.

[0147] A determining unit configured to perform weighted summation on a prediction loss between the first dictionary encoding prediction result and a second dictionary encoding prediction result and the reference loss to obtain the loss value.

[0148] In some embodiments, the reference loss comprises at least one of:

[0149] A first loss between the lip features of the lip sample sequence and text features of the target text.

[0150] A second loss between a phoneme prediction result of the lip sample sequence and a phoneme index sequence of the target text.

[0151] In some embodiments, the determining module is specifically configured to:

[0152] Obtain the lip features obtained by processing the lip sample sequence by a lip encoder in the lip processing model; and

[0153] Obtain the text features by processing the target text based on a text feature extractor.

[0154] Determine the first loss between the lip features and the text features.

[0155] In some embodiments, the determining module is specifically configured to:

[0156] input the mouth shape feature into a phoneme recognition network for recognizing phonemes, to obtain a phoneme prediction result of the mouth shape sample sequence;

[0157] compare the phoneme prediction result with a phoneme index sequence of the target text, to obtain the second loss.

[0158] In some embodiments, the phoneme recognition network comprises a phoneme-level regression network and a phoneme classifier.

[0159] The phoneme-level regression network is configured to process the mouth shape feature, to obtain a phoneme feature.

[0160] The phoneme classifier is configured to process the phoneme feature, to predict the phoneme prediction result of the mouth shape sample sequence.

[0161] In some embodiments, the weights of the reference loss and the prediction loss are obtained by training the mouth shape processing model.

[0162] In some embodiments, the method further comprises an optimization unit configured to, in a case where a new word is added to a dictionary, continue to optimize the mouth shape interpretation model on the basis of model parameters of the mouth shape interpretation model, so that the mouth shape interpretation model is applicable to the new word.

[0163] Based on the same technical concept, the embodiments of the present disclosure also provide a speech recognition apparatus 900 applied to a mouth shape interpretation model trained by the model training apparatus, as shown in Figure 9 The speech recognition apparatus 900 comprises:

[0164] An acquisition module 901 is configured to acquire a target speech of a target object and a mouth shape image sequence corresponding to the target speech.

[0165] A first prediction module 902 is configured to process the mouth shape image sequence based on the mouth shape interpretation model, to obtain a third dictionary encoding prediction result; and

[0166] A second prediction module 903 is configured to input the target speech into a target speech recognition network, to obtain a fourth dictionary encoding prediction result.

[0167] A fusion module 904 is configured to fuse the third dictionary encoding prediction result and the fourth dictionary encoding prediction result, to obtain a fused encoding prediction result.

[0168] An identification module 905 is configured to parse the fused encoding prediction result based on a dictionary, to obtain text information corresponding to the target speech.

[0169] In some embodiments, the fusion module comprises:

[0170] A first fusion unit is configured to perform a point multiplication operation on the same bits in the third dictionary coding prediction result and the fourth dictionary coding prediction result to obtain the fusion coding prediction result.

[0171] In some embodiments, the fusion module comprises, comprising:

[0172] A processing unit is configured to perform an accumulation operation on the same bits in the third dictionary coding prediction result and the fourth dictionary coding prediction result to obtain an intermediate coding result.

[0173] A second fusion unit is configured to convert the intermediate coding result into a probability distribution to obtain the fusion coding prediction result.

[0174] In some embodiments, the acquisition module comprises:

[0175] An acquisition unit is configured to acquire a video of the target object acquired synchronously with the target voice.

[0176] A sampling unit is configured to perform frame extraction on the video based on a target sampling rate to obtain the lip shape image sequence.

[0177] In some embodiments, the first prediction module comprises:

[0178] An extraction unit is configured to extract a lip shape feature point from each frame of image in the lip shape image sequence.

[0179] A normalization unit is configured to normalize the distance between the lip shape feature points to obtain a mouth opening coefficient sequence.

[0180] An analysis unit is configured to input the mouth opening coefficient sequence into the lip shape interpretation model to obtain a third dictionary coding prediction result.

[0181] The specific functions and examples of the modules and sub-modules of the apparatuses in the embodiments of the present disclosure are described above in the corresponding steps of the method embodiments, and will not be described here.

[0182] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0183] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0184] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0185] As shown in Figure 10 The device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded into a random access memory (RAM) 1003 from a storage unit 1008. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0186] Various components in the device 1000 are connected to the I / O interface 1005, including an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; the storage unit 1008, such as magnetic disks, optical disks, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through computer networks, such as the Internet, and / or various telecommunication networks.

[0187] The computing unit 1001 can be various general purpose and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs various methods and processes described above, such as the model training method and / or the speech recognition method. For example, in some embodiments, the model training method and / or the speech recognition method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded onto the RAM 1003 and executed by the computing unit 1001, one or more steps of the model training method and / or the speech recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the model training method and / or the speech recognition method by any other suitable means, such as by means of firmware.

[0188] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0189] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0190] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0191] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0192] In addition, an embodiment of the present disclosure further provides a vehicle, comprising the electronic device described above.

[0193] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0194] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0195] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the steps described above. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0196] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model training method, comprising: inputting a lip shape sample sequence into a lip shape processing model to obtain a first dictionary code prediction result predicted based on the lip shape sample sequence; determining a loss value based on the first dictionary code prediction result and a second dictionary code prediction result, the second dictionary code prediction result being determined based on target text corresponding to the lip shape sample sequence; adjusting model parameters of the lip shape processing model based on the loss value to obtain a lip shape interpretation model, wherein the lip shape interpretation model is used to assist speech recognition.

2. The method of claim 1, wherein, The lip shape processing model comprises a lip shape encoder, a lip shape decoder, and a word classifier connected after the lip shape decoder. The lip shape encoder is configured to process the lip shape sample sequence to obtain lip shape features. The lip shape decoder is configured to process the lip shape features to obtain intermediate features. The word classifier is configured to process the intermediate features to predict the first dictionary code prediction result.

3. The method of claim 1 or 2, wherein, The determination of the loss value based on the first dictionary code prediction result and the second dictionary code prediction result comprises: determining a reference loss; weighting and summing a prediction loss between the first dictionary code prediction result and the second dictionary code prediction result and the reference loss to obtain the loss value.

4. The method of claim 3, wherein, The reference loss comprises at least one of: a first loss between the lip shape features of the lip shape sample sequence and text features of the target text; a second loss between a phoneme prediction result of the lip shape sample sequence and a phoneme index sequence of the target text.

5. The method of claim 4, wherein, The determination of the first loss comprises: obtaining the lip shape features obtained by processing the lip shape sample sequence by the lip shape encoder in the lip shape processing model; and obtaining the text features by processing the target text based on a text feature extractor; determining the first loss between the lip shape features and the text features.

6. The method of claim 4, wherein, The determination of the second loss comprises: inputting the lip shape features into a phoneme recognition network for recognizing phonemes to obtain a phoneme prediction result of the lip shape sample sequence; comparing the phoneme prediction result with a phoneme index sequence of the target text to obtain the second loss.

7. The method of claim 6, wherein, The phoneme recognition network comprises a phoneme-level regression network and a phoneme classifier. The phoneme-level regression network is configured to process the lip shape features to obtain phoneme features. The phoneme classifier is configured to process the phoneme features to predict the phoneme prediction result of the lip shape sample sequence.

8. The method of claim 3, wherein, The weights of the reference loss and the prediction loss are obtained by training the lip shape processing model.

9. The method of claim 1, wherein, In the case of adding new words in the dictionary, the lip shape interpretation model is further optimized based on the model parameters of the lip shape interpretation model to adapt the lip shape interpretation model to the new words.

10. A speech recognition method applied to a lip shape interpretation model trained by the method of any one of claims 1-9, comprising: obtaining target speech of a target object and a lip shape image sequence corresponding to the target speech; processing the lip shape image sequence based on the lip shape interpretation model to obtain a third dictionary code prediction result; and input the target speech into a target speech recognition network to obtain a fourth dictionary code prediction result; fuse the third dictionary code prediction result and the fourth dictionary code prediction result to obtain a fused code prediction result; parse the fused code prediction result based on a dictionary to obtain text information corresponding to the target speech.

11. The method of claim 10, wherein, The fusing the third dictionary code prediction result and the fourth dictionary code prediction result to obtain a fused code prediction result comprises: point-multiplying the same bits in the third dictionary code prediction result and the fourth dictionary code prediction result to obtain the fused code prediction result.

12. The method of claim 10, wherein, The fusing the third dictionary code prediction result and the fourth dictionary code prediction result to obtain a fused code prediction result comprises: performing accumulation operation on the same bits in the third dictionary code prediction result and the fourth dictionary code prediction result to obtain an intermediate code result; converting the intermediate code result into a probability distribution to obtain the fused code prediction result.

13. The method of claim 10, wherein, The obtaining the target speech of the target object and a sequence of lip shape images corresponding to the target speech comprises: obtaining a video of the target object acquired synchronously with the target speech; based on a target sampling rate, frame extraction is performed on the video to obtain the sequence of lip shape images.

14. The method of claim 10, wherein, The processing the sequence of lip shape images based on the lip shape interpretation model to obtain a third dictionary code prediction result comprises: extracting lip shape feature points from each frame of image in the sequence of lip shape images; normalizing the distances between the lip shape feature points to obtain a sequence of mouth opening coefficients; inputting the sequence of mouth opening coefficients into the lip shape interpretation model to obtain the third dictionary code prediction result.

15. A model training apparatus, comprising: an input module configured to input a sequence of lip shape samples into a lip shape processing model to obtain a first dictionary code prediction result predicted based on the sequence of lip shape samples; a determination module configured to determine a loss value based on the first dictionary code prediction result and a second dictionary code prediction result, the second dictionary code prediction result being determined based on target text corresponding to the sequence of lip shape samples; an optimization module configured to adjust model parameters of the lip shape processing model based on the loss value to obtain a lip shape interpretation model, wherein the lip shape interpretation model is configured to assist speech recognition.

16. The apparatus of claim 15, wherein, The lip shape processing model comprises a lip shape encoder, a lip shape decoder, and a word classifier connected after the lip shape decoder. The lip shape encoder is configured to process the sequence of lip shape samples to obtain lip shape features. The lip shape decoder is configured to process the lip shape features to obtain intermediate features. The word classifier is configured to process the intermediate features to predict the first dictionary code prediction result.

17. The apparatus of claim 15 or 16, wherein, The determination module comprises: an auxiliary unit configured to determine a reference loss; a determination unit configured to perform weighted summation on a prediction loss between the first dictionary code prediction result and the second dictionary code prediction result and the reference loss to obtain the loss value.

18. The apparatus of claim 17, wherein, The reference loss comprises at least one of: a first loss between the lip shape features of the sequence of lip shape samples and text features of the target text; a second loss between a phoneme prediction result of the mouth shape sample sequence and a phoneme index sequence of the target text.

19. The apparatus of claim 18, wherein, The determining module is specifically configured to: obtain the mouth shape feature obtained by processing the mouth shape sample sequence by a mouth shape encoder in the mouth shape processing model; and obtain the text feature based on processing the target text by a text feature extractor; determine the first loss between the mouth shape feature and the text feature.

20. The apparatus of claim 18, wherein, The determining module is specifically configured to: input the mouth shape feature into a phoneme recognition network for recognizing phonemes to obtain a phoneme prediction result of the mouth shape sample sequence; compare the phoneme prediction result with a phoneme index sequence of the target text to obtain the second loss.

21. The apparatus of claim 20, wherein, The phoneme recognition network comprises a phoneme-level regression network and a phoneme classifier; The phoneme-level regression network is configured to process the mouth shape feature to obtain a phoneme feature; The phoneme classifier is configured to process the phoneme feature to predict the phoneme prediction result of the mouth shape sample sequence.

22. The apparatus of claim 17, wherein, The weights of the reference loss and the prediction loss are obtained by training the mouth shape processing model.

23. The apparatus of claim 15, further comprising an optimization unit configured to, in a case where a new word is added in a dictionary, continue to optimize the mouth shape interpretation model based on model parameters of the mouth shape interpretation model, so that the mouth shape interpretation model is applicable to the new word.

24. A speech recognition apparatus, applied to a mouth shape interpretation model trained by the apparatus of any one of claims 15-23, comprising: an obtaining module configured to obtain a target speech of a target object and a mouth shape image sequence corresponding to the target speech; a first predicting module configured to process the mouth shape image sequence based on the mouth shape interpretation model to obtain a third dictionary code prediction result; and a second predicting module configured to input the target speech into a target speech recognition network to obtain a fourth dictionary code prediction result; a fusing module configured to fuse the third dictionary code prediction result and the fourth dictionary code prediction result to obtain a fused code prediction result; an identifying module configured to parse the fused code prediction result based on a dictionary to obtain text information corresponding to the target speech.

25. The apparatus of claim 24, wherein, The fusing module comprises: a first fusing unit configured to perform a point multiplication operation on same bits in the third dictionary code prediction result and the fourth dictionary code prediction result to obtain the fused code prediction result.

26. The apparatus of claim 24, wherein, The fusing module comprises: a processing unit configured to perform an accumulation operation on same bits in the third dictionary code prediction result and the fourth dictionary code prediction result to obtain an intermediate code result; a second fusing unit configured to convert the intermediate code result into a probability distribution to obtain the fused code prediction result.

27. The apparatus of claim 24, wherein, The obtaining module comprises: an obtaining unit configured to obtain a video of the target object obtained synchronously with the target speech; a sampling unit configured to perform frame extraction on the video based on a target sampling rate to obtain the mouth shape image sequence.

28. The apparatus of claim 24, wherein, The first predicting module comprises: an extracting unit configured to extract a mouth shape feature point from each frame of image in the mouth shape image sequence; a normalization unit, configured to normalize distances between the mouth feature points to obtain a mouth opening coefficient sequence; a parsing unit, configured to input the mouth opening coefficient sequence into the mouth shape interpretation model to obtain a third dictionary encoding prediction result. 29.An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

30. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method according to any one of claims 1-14. 31.A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-14. 32.A vehicle comprising the electronic device of claim 29.

Images