Speech recognition method, model training method, device, electronic equipment and vehicle

Abstract

The present application relates to a kind of speech recognition method, the training method of model, device, electronic equipment and vehicle, including the feature extraction of mixed audio stream, obtain the first audio feature of target object and the second audio feature of other object except the target object, including the speech of at least two objects in mixed audio stream, target object is distinguished from at least two objects One object;According to first audio feature, obtain the first feature vector of the speech emitted by target object;According to second audio feature, obtain the second feature vector associated with the speech possibly emitted by target object by prediction;According to first feature vector and second feature vector, the target text corresponding to the speech emitted by target object in mixed audio stream is identified to obtain.This embodiment of the present application can improve the recognition accuracy of mixed audio stream by the feature of double-path extraction to identify the speech of target object in mixed audio stream.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech recognition method, a model training method, a device, an electronic device and a vehicle. BACKGROUND

[0002] Speech recognition refers to a human-computer interaction process of converting human language into text or commands. At present, speech recognition technology is widely used in scenarios such as voice interaction, voice transcription, voice analysis, audio and video recording.

[0003] In related technologies, speech recognition is performed by using a deep learning model to train a large amount of data to learn the relationship between an audio signal and corresponding text, so as to convert the user's speech into text. In human-computer interaction scenarios, the speech collected is generally mixed speech (the sound of multiple people speaking or environmental noise), so the recognition of mixed speech streams is more extensive. The deep learning model in related technologies processes mixed speech streams in the following manner: continuously extracts speech features emitted by a single object from the mixed speech stream, and then identifies the target text corresponding to the speech features. This speech recognition manner has low accuracy. For mixed speech stream recognition, the model needs to continuously recognize, and as the length of the speech stream increases, the recognition accuracy gradually decreases. Therefore, after recognizing a mixed speech stream of a certain length, the model needs to pause for a period of time before starting to recognize again, so the speech recognition manner in related technologies has low accuracy and needs frequent pauses.

[0004] Therefore, there is an urgent need to provide a speech recognition method with higher recognition accuracy to reduce pauses. SUMMARY

[0005] One of the purposes of the present application is to provide a speech recognition method to solve the problem of low recognition accuracy of mixed speech in the prior art. The second purpose is to provide a speech recognition model training method. The third purpose is to provide a speech recognition device. The fourth purpose is to provide a speech recognition model training device. The fifth purpose is to provide an electronic device. The sixth purpose is to provide a vehicle.

[0006] To achieve the above purposes, the technical solutions adopted by the present application are as follows:

[0007] A speech recognition method, the method comprising:

[0008] performing feature extraction on a mixed audio stream to obtain first audio features of a target object and second audio features of other objects except the target object, the mixed audio stream including speech of at least two objects, the target object being one object distinguished from the at least two objects;

[0009] obtain a first feature vector of the voice emitted by the target object according to the first audio feature;

[0010] obtain a second feature vector associated with the voice possibly emitted by the target object according to the second audio feature;

[0011] identify the target text corresponding to the voice emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

[0012] According to the above technical means, when extracting features from the mixed audio stream, the features of the mixed audio stream are extracted as first audio features and second audio features, the mixed audio stream is distinguished, and then divided into two processing paths, and the first audio features and the second audio features are processed respectively, wherein the first audio features are processed to obtain a first feature vector as a first path of feature processing; the second audio features are processed to obtain a second feature vector as a second path of feature processing, and finally the target text is identified by fusing the first feature vector and the second feature vector. First, the mapping probability of the first audio feature and the target text is calculated through the first path, and the mapping probability of the first audio feature and the target text is predicted through the second path, and the prediction results of the two paths are fused, which can improve the accuracy of speech recognition. Or it can be understood that in the same voice interaction scene, the voice of other objects in the mixed voice stream and the voice of the target object have certain correlation, and the embodiment predicts the mapping probability of the voice of the target object and the target text by adding the second feature vector associated with the voice possibly emitted by the target object, which can improve the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scene of the mixed audio stream, the speech recognition accuracy is high, and the speech recognition pause in the recognition process can be reduced.

[0013] Further, the identifying the target text corresponding to the voice emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector comprises:

[0014] time-synchronously fusing the first feature vector and the second feature vector to obtain a fused feature vector;

[0015] identifying the target text corresponding to the voice emitted by the target object in the mixed audio stream according to the fused feature vector.

[0016] According to the above technical means, by converting the first audio feature into a first feature vector and the second audio feature into a second feature vector, and then fusing, the accuracy is higher and the calculation amount is reduced, which provides the possibility for deploying the recognition method on the end side.

[0017] Further, the identification method further comprises:

[0018] predicting a backward probability of speech possibly emitted by the target object at a next time step according to the first audio feature;

[0019] The identification of the target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector comprises:

[0020] The target text corresponding to the speech emitted by the target object in the mixed audio stream is predicted by a preset conditional probability based on the first feature vector, the second feature vector, the backward probability and a forward probability of the speech emitted by the target object predicted at a previous time step.

[0021] According to the above technical means, the forward probability and the backward probability associate the features before and after the speech sequence, and for the recognition of the streaming audio feature, the features of the words or sentences are composed based on the features before and after, and the target text corresponding to the speech emitted by the target object is predicted based on the forward probability and the backward probability, which can further improve the recognition accuracy and reduce the frequency of the speech recognition model based on the recognition accuracy reduction.

[0022] Further, the feature extraction of the mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects except the target object comprises:

[0023] The mixed audio stream is preprocessed to obtain mixed audio features;

[0024] The mask of the speech emitted by the target object is predicted according to the mixed audio features;

[0025] The first audio feature of the target object is obtained according to the mask and the mixed audio features;

[0026] The second audio feature of other objects except the target object is obtained according to the complement of the mask and the mixed audio features.

[0027] According to the above technical means, the mixed audio stream is preprocessed to extract mixed audio features conforming to the format, which is convenient for mask processing. By mask processing the mixed audio stream, the features corresponding to the speech of different objects in the mixed audio stream are distinguished, and the mixed audio stream is separated into at least two paths corresponding to the required audio features.

[0028] Further, the application is applied to a computing device deployed with a speech recognition model, and the speech recognition model comprises a mask network, a long short-term memory network, a prediction network and a joint network, wherein:

[0029] The mask network is configured to perform feature extraction on the mixed audio stream to obtain first audio features of a target object and second audio features of other objects except the target object.

[0030] The long short-term memory network is configured to obtain a first feature vector of speech emitted by the target object according to the first audio features.

[0031] The prediction network is configured to obtain a second feature vector associated with speech possibly emitted by the target object according to the second audio features.

[0032] The joint network is configured to identify target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

[0033] According to the above technical means, since the mask network, the long short-term memory network, and the prediction network in the speech recognition model all perform feature extraction on audio features, and the joint network directly maps the probability of text after fusing the features to obtain target text, the end-to-end streaming speech recognition is realized after the mixed audio stream is input into the speech recognition model, and the corresponding target text is directly output without multiple cascades, thereby improving the recognition accuracy.

[0034] Further, after the target text corresponding to the speech emitted by the target object in the mixed audio stream is identified, the following steps are further included.

[0035] When it is monitored that the target text corresponding to the speech emitted by the target object at the current moment is abnormal, the model parameters of the speech recognition model are optimized based on the abnormal scenario of the target text, the speech recognition application field, and the corrected text, wherein the corrected text is the correct text after correction corresponding to the abnormal target text.

[0036] According to the above technical means, in the process of identifying speech, the abnormal scenario of speech recognition is monitored, the reasons and types causing the recognition abnormality in the process of speech recognition are learned, and the learning is associated with the language recognition application field. Through continuous optimization of the model parameters of the speech recognition model in the abnormal scenario, the model gradually adapts to the scenario that may appear abnormal, reduces the situation of recognition abnormality of the speech recognition model, and further makes the accuracy of model recognition higher and higher. When enough abnormal scenarios of the speech recognition application field are learned, non-stop streaming speech recognition can be realized. That is, based on the above technical means, the speech recognition model adapts to the dynamically changing indefinite-length data stream, and non-stop streaming recognition is realized.

[0037] Further, the optimization of the model parameters of the speech recognition model based on the abnormal scenario of the target text, the speech recognition application field, and the corrected text includes:

[0038] adjust the loss of the speech recognition model according to the abnormal scene of the target text, the speech recognition application field, and the correction text;

[0039] adjust the forward probability algorithm model parameter and the backward probability algorithm model parameter based on the adjusted loss;

[0040] predict the forward probability of the speech that the target object may emit at a previous moment of the current moment according to the adjusted forward probability algorithm model parameter, and predict the backward probability of the speech that the target object may emit at a next moment of the current moment based on the adjusted backward probability algorithm model parameter;

[0041] calculate the gradient according to the forward probability, the backward probability, and the correction text;

[0042] adjust the model parameter of the speech recognition model based on the gradient.

[0043] According to the above technical means, the forward probability algorithm model parameter and the backward probability algorithm model parameter can be adjusted, and then the gradient is calculated to update the model parameter, which can optimize the model parameter on the one hand, and quickly find the model parameter with the minimum loss on the other hand.

[0044] Further, the model parameter of the speech recognition model is optimized through the cloud.

[0045] According to the above technical means, the model parameter of the speech recognition model is optimized in the cloud, and the OTA can be realized based on the cloud and the end, and the algorithm power of the end can be reduced, so that the normal use of other functions of the end is avoided when the model parameter is optimized.

[0046] A training method of a speech recognition model, comprising:

[0047] obtain a training sample, the training sample comprising a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream;

[0048] input the sample mixed audio stream into a speech recognition model, and perform the following processing in the speech recognition model to obtain a target text corresponding to the speech emitted by a target object in the sample mixed audio stream: extract features from the sample mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object; obtain a first feature vector of the speech emitted by the target object according to the first audio features, and predict a second feature vector associated with the speech that the target object may emit according to the second audio features; and identify the target text according to the first feature vector and the second feature vector;

[0049] calculate a loss value of the target text relative to a corresponding target sample label based on a total loss function;

[0050] adjust model parameters of the speech recognition model according to the loss value.

[0051] According to the above technical means, by setting the speech recognition model to extract the mixed audio stream as first audio features and second audio features during feature extraction, the mixed audio stream is distinguished and then divided into two processing paths, and the first audio features and the second audio features are processed respectively, wherein the first audio features are processed to obtain the first feature vector as the first path of feature processing; the second audio features are processed to obtain the second feature vector as the second path of feature processing, and the first feature vector and the second feature vector are fused to finally recognize the target text. First, the mapping probability of the first audio features and the target text is calculated through the first path, and the mapping probability of the first audio features and the target text is predicted through the second path, and the prediction results of the two paths are fused, which can improve the accuracy of speech recognition. Or it can be understood that in the same speech interaction scene, the speech of other objects in the mixed speech stream and the speech of the target object have a certain correlation, and the embodiment predicts the mapping probability of the speech of the target object and the target text by adding the second feature vector associated with the speech of the target object, which can improve the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scene of the mixed audio stream, the speech recognition accuracy is high, and the speech recognition pause situation in the recognition process can be reduced.

[0052] Further, the speech recognition model includes a mask network, and the feature extraction of the sample mixed audio stream to obtain the first audio features of the target object and the second audio features of other objects other than the target object includes:

[0053] inputting the sample mixed audio stream into a pre-processing encoder of the mask network, pre-processing the sample mixed audio stream based on the pre-processing encoder to obtain sample mixed audio features;

[0054] inputting the sample audio features into a mask encoder of the mask network, processing the sample audio features based on the mask encoder to predict a mask of the speech of the target object;

[0055] obtaining the first audio features of the target object according to the mask and the sample audio features;

[0056] obtaining the second audio features of other objects other than the target object according to the mask corresponding complement and the sample audio features.

[0057] According to the above technical means, by performing mask processing on the mixed audio stream, the features corresponding to the speech of different objects in the mixed audio stream are distinguished, and the mixed audio stream is separated into at least two paths corresponding to the required audio features.

[0058] Further, the speech recognition model comprises a long short-term memory network, and the first feature vector of the speech emitted by the target object is obtained according to the first audio feature, comprising:

[0059] The first audio feature is input into the long short-term memory network, the long short-term memory network is used for feature extraction on the first audio feature, and the first feature vector of the speech emitted by the target object is obtained;

[0060] And / or, the speech recognition model comprises a prediction network, and the second feature vector of the speech possibly emitted by the target object is predicted according to the second audio feature, comprising:

[0061] The second audio feature is input into the prediction network, the prediction network is used for predicting the speech possibly emitted by the target object corresponding to the second audio feature, and the corresponding second feature vector is generated.

[0062] According to the above technical means, the training of the speech recognition model can obtain that after the mixed audio stream is input into the speech recognition model, the audio feature is extracted directly based on the mixed audio stream, then the corresponding target text is output, without multiple cascades, realizing end-to-end speech recognition, and improving the recognition accuracy.

[0063] Further, the speech recognition model comprises a joint network, and the target text is identified according to the first feature vector and the second feature vector, comprising:

[0064] The first feature vector and the second feature vector are fused based on the joint network to obtain a fusion feature vector;

[0065] According to the fusion feature vector, the target text corresponding to the speech emitted by the target object in the sample mixed audio stream is identified.

[0066] According to the above technical means, the association between the fusion of the first feature vector and the second feature vector and the text is learned, and the speech recognition accuracy is improved.

[0067] Further, the training method further comprises:

[0068] The backward probability of the speech possibly emitted by the target object at the next time step is predicted according to the first audio feature;

[0069] The target text is identified according to the first feature vector and the second feature vector, and the method comprises:

[0070] The target text is predicted by using a preset conditional probability based on the first feature vector, the second feature vector, the backward probability, and a forward probability of the speech emitted by the target object predicted at a previous time step.

[0071] According to the above technical means, the correlation between the predicted probability of the training sample at the current time step and the target sample label and the forward probability and the backward probability is learned based on the forward probability of the training sample at the previous time step corresponding to the same sequence label and the backward probability of the training sample at the next time step corresponding to the same sequence label, so that the loss of the target text and the target sample label calculated is updated, and the accuracy of speech recognition is improved.

[0072] Further, the speech recognition model further comprises an audio encoder and a label encoder, and the training method further comprises:

[0073] The first audio feature is input into the audio encoder, the first audio feature is encoded based on the audio encoder, and the sample label is encoded according to the label encoder;

[0074] The backward probability of the speech that the target object may emit at the next time step is predicted according to the first audio feature;

[0075] The gradient of the loss function with respect to the model parameter is predicted according to the backward probability and the forward probability of the speech emitted by the target object predicted at the previous time step.

[0076] The model parameter of the speech recognition model is adjusted according to the loss value, and the method comprises:

[0077] When the loss value is greater than or equal to a preset threshold, the model parameter is adjusted according to the gradient.

[0078] According to the above technical means, the gradient reflects the sensitivity (partial derivative of the model parameter) of the difference (loss value) between the target text recognized by the speech recognition model and the sample label with respect to the change of the model parameter. By calculating the gradient, the model parameter is adjusted based on the gradient, so that the speech recognition model can quickly learn the correlation between the mixed audio stream and the sample label, reduce the learning samples, and improve the learning efficiency. In addition, the gradient is calculated based on the forward probability and the backward probability, and the gradient guides the model parameter, so that the difference between the target text recognized by the speech recognition model and the sample label tends to be smaller.

[0079] Further, the training method further comprises:

[0080] The total loss function is constructed based on errors of target text and target sample labels predicted by the first audio features and errors of target text and target sample labels predicted by the second audio features.

[0081] A speech recognition device comprises:

[0082] An extraction module is configured to perform feature extraction on the mixed audio stream to obtain first audio features of a target object and second audio features of other objects except the target object.

[0083] A first prediction module is configured to obtain a first feature vector of speech uttered by the target object based on the first audio features.

[0084] A second prediction module is configured to predict a second feature vector associated with speech possibly uttered by the target object based on the second audio features.

[0085] A recognition processing module is configured to recognize target text corresponding to the speech uttered by the target object in the mixed audio stream based on the first feature vector and the second feature vector.

[0086] A training device of a speech recognition model comprises:

[0087] An acquisition module is configured to acquire training samples, wherein the training samples comprise sample mixed audio streams and sample labels corresponding to the sample mixed audio streams.

[0088] A speech recognition model is configured to perform the following processing on the input sample mixed audio streams to recognize target text corresponding to speech uttered by a target object in the sample mixed audio streams: performing feature extraction on the sample mixed audio streams to obtain first audio features of the target object and second audio features of other objects except the target object; obtaining a first feature vector of speech uttered by the target object based on the first audio features; predicting a second feature vector associated with speech possibly uttered by the target object based on the second audio features; and recognizing the target text based on the first feature vector and the second feature vector.

[0089] A training module is configured to calculate a loss value of the target text relative to the corresponding sample labels based on a total loss function and adjust model parameters of the speech recognition model based on the loss value.

[0090] An electronic device comprises a memory and a processor.

[0091] The memory is configured to store computer programs / instructions, and the processor is configured to implement the method described in the above item based on the computer programs / instructions stored in the memory.

[0092] A vehicle, comprising: a memory, a processor;

[0093] The memory is used for storing computer programs / instructions; the processor is used for realizing the method as described above according to the computer programs / instructions stored in the memory.

[0094] A computer readable storage medium / computer program product, the computer readable storage medium stores computer programs / instructions, or the computer program product comprises computer programs / instructions, the computer programs / instructions are executed by a processor to realize the method as described above.

[0095] The beneficial effects of the present application are:

[0096] (1) The present application extracts audio features by setting double paths, one path extracts first audio feature quantities through the first audio features of the target object, and the other path predicts the second feature vector corresponding to the voice possibly emitted by the target object through the second audio features of other objects, and identifies the voice emitted by the target object through the two feature vectors, and the prediction and fusion of the two features can improve the accuracy of identifying the voice emitted by the target object in the mixed audio stream.

[0097] (2) The present application realizes the mapping of acoustic feature sequences to label sequences through a speech recognition model, and synchronously converts, without a multi-cascade model, to simplify the model.

[0098] (3) The present application calculates the partial derivative (gradient) of the loss function of the forward probability and backward probability calculation model with respect to the model parameters, which is used to guide how to adjust the model parameters to minimize the loss function, so that the speech recognition model has small speech recognition error; combined with a heuristic error distribution method, it can process audio streams of indefinite length and can reduce or even avoid the occurrence of lag.

[0099] (4) The system resources (CPU (processor), RAM (memory), EMMC (Flash storage)) consume less and have high accuracy, so that the speech recognition model is easy to deploy on the end side. BRIEF DESCRIPTION OF DRAWINGS

[0100] Figure 1 A hardware system example diagram of the speech recognition method provided by the embodiment of the present application;

[0101] Figure 2 An application scenario diagram of the speech recognition method provided by the embodiment of the present application;

[0102] Figure 3 A software system diagram of the speech recognition method provided by the embodiment of the present application;

[0103] Figure 4An embodiment flowchart of a voice recognition method provided by the embodiment of the present application is shown in the following.

[0104] Figure 5 An embodiment flowchart of a voice recognition method provided by the embodiment of the present application is shown in the following.

[0105] Figure 6 An embodiment flowchart of a voice recognition method provided by the embodiment of the present application is shown in the following.

[0106] Figure 7 An embodiment flowchart of a voice recognition method provided by the embodiment of the present application is shown in the following.

[0107] Figure 8 An embodiment flowchart of a voice recognition method provided by the embodiment of the present application is shown in the following.

[0108] Figure 9 An embodiment flowchart of a voice recognition method provided by the embodiment of the present application is shown in the following. DETAILED DESCRIPTION

[0109] Other advantages and effects of the present application can be easily understood by those skilled in the art from the contents disclosed in the present specification. The present application can also be implemented or applied in other different specific embodiments, and various modifications or changes can be made to the details in the present specification based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, but not for limiting the protection scope of the present application.

[0110] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the diagrams, but not the number, shape and size of the components when actually implemented. The shape, number and ratio of the components when actually implemented can be arbitrarily changed, and the layout pattern of the components can also be more complex.

[0111] Voice recognition refers to the process of converting human language into text or commands for human-machine interaction. Currently, voice recognition technology is widely used in voice interaction, voice transcription, voice analysis, audio and video recording, etc.

[0112] In the related art, the recognition of voice is achieved by using a deep learning model to train a large amount of data and learn the relationship between the audio signal and the corresponding text, so as to convert the user's voice into text.

[0113] However, in a human-computer interaction scene, generally collected speech is mixed speech (the sound of multiple people speaking or environmental noise), and the mixed speech affects the accuracy of recognition, and the recognition accuracy of the above speech recognition manner is low.

[0114] Based on this, an embodiment of the present application provides a speech recognition method, which recognizes speech through double-path feature prediction results, wherein one path is based on audio features of a target object for recognition, and the other path is based on other audio features of other objects in mixed sound except the target object for prediction, which can improve the accuracy of recognition.

[0115] The following describes the hardware system, software system and specific implementation embodiments involved in the application of an embodiment of the present application.

[0116] Figure 1 A hardware system example diagram of the speech recognition method provided by the embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the hardware system is applied in a device end, and the device end includes a microphone MIC, a signal processor, a low-pass filter, an analog-to-digital conversion ADC / A2B bus, an audio digital signal processor Audio DSP, a master controller AVN / HU / IVI SOC, and a communication module, etc.

[0117] The microphone MIC is used to collect an audio signal (for streaming speech, it corresponds to collecting an audio signal stream). The audio signal collected by the microphone MIC is transmitted to the signal processor, and the signal processor performs gain amplification (for example, an active filter amplifier or a program-controlled amplifier) and noise reduction processing on the audio signal. After the signal processor performs gain amplification and noise reduction processing on the audio signal, the audio signal is transmitted to the low-pass filter, and the low-pass filter performs low-pass filtering on the audio signal to remove high-frequency signals. The low-pass filter performs analog-to-digital conversion on the audio signal after removing high-frequency signals through the analog-to-digital conversion ADC / A2B bus to convert the audio signal into an audio digital signal, and the audio digital signal is input to the Audio DSP for processing.

[0118] Optionally, in the embodiment, the low-pass filter filters audio signals exceeding 20KHz high frequency.

[0119] Optionally, the above hardware system is used to process the audio signal.

[0120] As an example, the speech recognition method is applied in the application scenario of an intelligent network car, and the hardware system is arranged on the car. As shown in FIG. 2, the hardware system is arranged in the car. Figure 2As shown, the microphone MIC is arranged in the intelligent cabin to collect audio signals of people speaking in the intelligent cabin. When applied to a vehicle, the driver or passenger can use voice recognition interaction, voice transcription, voice analysis, audio and video recording and other functions inside the vehicle. Through voice interaction control, the driver can focus on driving while realizing voice control of the vehicle, thereby reducing the risk of accidents caused by distracted operation of the vehicle system and improving driving comfort. Therefore, the embodiments of the present application applied to a vehicle have better use effect.

[0121] Figure 3 A software system example of the voice recognition method provided by the embodiments of the present application is shown in Figure 3 As shown, the software system can be configured in the master controller of the device end. The software system includes a hardware abstraction layer (HAL layer), a service layer, a framework layer and an application layer, wherein:

[0122] The audio DSP performs digital noise reduction and data processing on the audio digital signal again, and sends the audio digital signal to the AVN / HU / IVI SOC (intelligent cabin master controller) system HAL (hardware abstraction layer) layer through the IIS (Inter-IC Sound) ( / PCM (Pulse Code Modulation) / TDM (Time Division Multiplexing) / PDM (Pulse Density Modulation) bus.

[0123] In the HAL layer, the audio digital signal is processed to recognize the text corresponding to the audio digital signal. Specifically, the speech feature sequence corresponding to the audio digital signal is extracted based on a double-path modeling algorithm, and the text sequence corresponding to the speech feature sequence is recognized based on the pre-learned association probability between the speech feature sequence and the text. The double-path modeling algorithm can improve the accuracy of speech recognition, especially for the case where the audio signal is mixed, the algorithm can distinguish the speech based on the other speech, and predict and recognize the speech distinguished based on the other speech, thereby improving the accuracy of recognition.

[0124] Further, in the HAL layer, the recognition of streaming speech is also realized based on a time-synchronized sequence conversion model.

[0125] The specific processing flow and implementation of the speech signal in the HAL layer can refer to the following embodiments.

[0126] Figure 4 A flowchart of a voice recognition method proposed by the embodiments is shown in Figure 4 The method includes:

[0127] S401, feature extraction is performed on the mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object;

[0128] The mixed audio stream is a streaming mixed audio collected by a microphone or a microphone array, which refers to continuously collected audio. The speech recognition method in the embodiments of the present application can process the mixed audio stream to realize real-time processing and feedback of the mixed audio.

[0129] After obtaining the mixed audio stream, feature extraction is performed on the mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object. The target object refers to a certain speaking object in the mixed audio stream, and the mixed audio stream includes speech of multiple objects, wherein the target object is an object distinguished from the multiple objects. It should be noted that all audio features are sorted in time sequence according to the time sequence of the input mixed audio stream, forming a time sequence, the first audio feature refers to an audio feature corresponding to the speech of the target object at a certain time step in the time sequence extracted from the mixed audio stream, the second audio feature refers to an audio feature corresponding to the speech of other objects at the time step, and the first audio feature and the second audio feature are audio features corresponding to the speech at the same time step.

[0130] As an example, features can be extracted from the mixed audio stream based on Mel frequency cepstral coefficients (MFCCs) to distinguish different sound sources, and the mixed audio stream is sequentially distinguished in time sequence, and the distinguished objects are sequentially taken as target objects.

[0131] For example, it is assumed that the mixed audio stream includes speech of two persons, A1 object and A2 object. If it is identified that the audio features of A1 object are emitted, it is determined that A1 object is the target object, the audio features corresponding to the speech of A1 object are extracted as the first audio features, and the audio features of other objects except A1 object are extracted as the second audio features. When the speech of A2 object is identified, A2 object is taken as the target object, the audio features corresponding to the speech of A2 object are extracted as the first audio features, and the audio features of other objects except A2 object are extracted as the second audio features. The speech of each object is sequentially distinguished to realize distinguishing the speech of each object from the mixed audio stream, and then the speech of each object is identified.

[0132] Suppose that the mixed audio stream includes speech of 3 or more persons, which are respectively A1 object, A2 object, …, An object. If the audio feature of the speech of the A1 object is recognized, it is determined that the A1 object is the target object, the audio feature of the speech of the A1 object is extracted as the first audio feature, and the audio features of the other objects (A2 to An) except the A1 object are extracted as the second audio features. At this time, if the second audio features are still mixed audio features, the mixed audio stream is updated based on the second audio features, and the speech in the mixed audio stream is continuously recognized.

[0133] As an example, refer to Figure 5 The speech recognition method in the embodiment of the application adopts an end-to-end speech recognition model for recognition. In this example, the speech recognition method is applied to a computing device in which the speech recognition model is deployed, wherein the speech recognition model includes a mask network, and wherein:

[0134] The mask network is configured to extract the first audio feature of the target object and the second audio feature of the other objects except the target object from the mixed audio stream.

[0135] The embodiment extracts the audio features of different objects through the mask network to distinguish the audio features of different objects. Optionally, the mask network includes a mask encoder, which can predict a mask from the mixed audio stream. The mask can reflect the contribution of different speakers in the mixed audio stream, and thus highlight the audio feature of one object and suppress the audio feature of another object. Therefore, the audio feature of one object can be extracted through the mask to distinguish the audio features of different objects.

[0136] Optionally, S401 can be specifically:

[0137] The mask of the speech of the target object is predicted from the mixed audio stream.

[0138] The mixed audio stream is processed according to the mask, the mixed audio stream, and the complement of the mask to obtain the first audio feature of the target object and the second audio feature of the other objects except the target object.

[0139] During training of the speech recognition model, the mask encoder learns to predict the encoding to distinguish each sound source in the mixed audio stream. Therefore, through the mask encoder, the mask of the speech of the target object can be predicted, and the first audio feature can be obtained by extracting the features of the mixed audio stream through the mask. The audio features other than the first audio feature are the second audio features.

[0140] For example, according to the mixed audio stream, a mask M corresponding to the voice emitted by the target object is predicted, and the mask value can be selected as [0, 1], and the complement of the mask is 1-M. The first audio feature corresponding to the target object is determined through the mask, and the specific first audio feature is H1, and then Correspondingly, the second audio feature corresponding to the object other than the target object is determined through the mask, and the specific second audio feature is H2, and then , wherein, is a mixed audio stream, or a mixed audio feature after preprocessing of the mixed audio stream, for example:

[0141] As an example, in order to further improve the accuracy of feature extraction, after obtaining the mixed audio stream, the mixed audio stream is preprocessed to obtain the mixed audio feature. The preprocessing includes converting the mixed audio stream into a format for further processing or a feature set.

[0142] Alternatively, in the example of applying to a speech recognition model, the mask network further includes a preprocessing encoder (MixEnc), and the mixed audio stream is processed by using the preprocessing encoder to obtain the mixed audio feature.

[0143] Optionally, the processing of the preprocessing encoder includes framing, window function, fast Fourier transform, power spectral density, mel filter bank, logarithmic compression, pooling, deep learning feature extraction, batch normalization, activation function, etc., to extract features and obtain the mixed audio feature.

[0144] As an example, the preprocessing encoder preprocesses the mixed audio stream through the following formula:

[0145] ;

[0146] Wherein, t is time; is continuous audio; is a window function; is the response of the mel filter to the frequency corresponding to the mel frequency ; ε is a constant; σ 2 is the variance; μ is the batch mean.

[0147] It should be noted that in the above formula, is an activation function; is a pooling function; log is a logarithm; STFT is a short-time Fourier transform (Short-Time Fourier Transform).

[0148] After preprocessing the mixed audio stream based on the above formula, MixkEnc(X) is obtained, and the corresponding mixed audio feature is:

[0149] ;

[0150] As an example, the mask encoder performs mask processing on the mixed audio feature X by the following formula:

[0151] ;

[0152] wherein, is the audio feature output by the layer convolution layer; is the input feature (audio feature) received by the mask encoder; is an activation function; is a convolutional neural network; is the weight of the fully connected layer is the bias of the fully connected layer. It should be noted that in the above formula, the MaskEnc convolutional neural network (CNN) is used to learn the feature representation of different speakers in the mixed audio stream.

[0153] Based on the above formula, after preprocessing the mixed audio stream, MaxkEnc(X) is obtained, and the prediction expression of the mask corresponding to the target object is:

[0154]

[0155] ; wherein, X is the mixed audio feature,

[0156] is a sigmoid activation function, which is used to map the output of MaskEnc to the interval (0, 1) to ensure that the element value of the mask is reasonable. The mask is a two-dimensional weight matrix. The first audio feature is obtained by element-wise multiplication of the mask and the mixed audio stream. The second audio feature is obtained by element-wise multiplication of the complement and the mixed audio stream. The first audio feature represents the audio feature of the target object, and the second audio feature represents the audio feature of other objects except the target object.

[0157] In this embodiment, when the mixed audio stream is preprocessed, the mixed audio stream is also time column encoded. The first audio feature obtained by the mask and the mixed audio stream and the second audio feature obtained by the complement and the mixed audio stream are time-synchronized features. This embodiment performs recognition based on the time-synchronized first audio feature and the second audio feature.

[0158] S402, according to the first audio feature, obtaining a first feature vector of the voice emitted by the target object;

[0159] S403, according to the second audio feature, obtaining a second feature vector of the voice emitted by the target object;

[0160] S404, according to the first feature vector and the second feature vector, obtaining a target object feature vector.In this step, the first audio feature is processed to obtain the association relationship between the first audio feature and the context, the dependency relationship with the text, and the time sequence, etc., to form a first feature vector, so that the corresponding text can be more accurately recognized through the first feature vector.

[0161] As an example, please refer to Figure 5 The speech recognition model further includes a long short-term memory network configured to obtain a first feature vector of the speech uttered by the target object according to the first audio feature. The long short-term memory network identifies, classifies, and processes the first audio feature in real time, captures the time dependency and change rule, models and predicts the first audio feature, and continuously and dynamically processes and identifies, to realize recognition of the streaming speech.

[0162] S403, according to the second audio feature, a second feature vector associated with the speech possibly uttered by the target object is predicted;

[0163] In this step, the second audio feature is predicted to predict the second feature vector associated with the speech possibly uttered by the target object. The second feature vector also includes the association relationship between the second audio feature and the context, the dependency relationship with the text, and the time sequence, etc.

[0164] The second audio feature is associated with the speech (first audio feature) possibly uttered by the target object: the first audio feature and the second audio feature are audio features corresponding to the same time step, and the speech uttered by different objects in the same scene has a certain association relationship. The embodiment defines the association relationship as an auxiliary role for speech recognition of the target object, which is a positive association feature. For example, in a conference scenario, different objects speak on the same topic, and the speech uttered in the same time step has an association relationship. The embodiment is applied to a mixed audio stream including the speech of A1 object and A2 object, and there is an association relationship based on the speaking content of A1 object and A2 object. The association relationship between the speaking of different objects in the mixed audio stream in the same scene can be set, the audio feature of A1 object is recognized as the first audio feature, the audio feature of A2 object is used to infer the speech possibly uttered by A1 object, and then the second feature vector corresponding to the speech is obtained.

[0165] For example, A1 object is the target object, and the text corresponding to the first audio feature may be "you". If the text corresponding to the second audio feature is 'hi', it is predicted that the speech possibly uttered by A1 object is "you", and the combination of the first feature vector and the second feature vector increases the recognition accuracy of the first audio feature. Similarly, if A2 is the target object, the feature processing is also performed in the above manner.

[0166] Or, in the mixed audio stream, including the voice of A1 object and A2 object, the text probability corresponding to the audio feature of A1 object is predicted, the text probability corresponding to the audio feature of A2 object is predicted, and then the text probability that A1 may emit is predicted by the audio feature of A2 object, the text probability of the final target text is predicted by combining the text probability predicted by the audio feature of A1 object and the text probability that A1 may emit predicted by A2 object. The prediction method in this example is similar or equivalent to the above example.

[0167] For example, A1 object is the target object, and the text corresponding to the first audio feature may be "you". If the text corresponding to the second audio feature is 'hi', then the combination of the first feature vector corresponding to the voice of "you" and the second feature vector corresponding to the voice of "hi" increases the recognition accuracy of the first audio feature. Similarly, if A2 is the target object, the feature processing is also performed in the above manner.

[0168] Alternatively, the association between the second audio feature and the voice (first audio feature) that the target object may emit can also be that the second audio feature can also be the background noise corresponding to the voice that the target object may emit. By identifying the background noise, the difference between the voice that the target object may emit is increased, so that the voice recognition accuracy is improved.

[0169] As an example, please refer to Figure 5 The voice recognition model further includes a prediction network configured to predict a second feature vector associated with the voice that the target object may emit according to the second audio feature. In the process of processing the second audio feature by the prediction network, the voice that the target object may emit at the same time step is predicted based on time sequence synchronization processing, and the voice recognition accuracy is improved.

[0170] S404, according to the first feature vector and the second feature vector, the voice corresponding to the target text emitted by the target object in the mixed audio stream is identified.

[0171] As an example, the first text prediction probability corresponding to the first feature vector can be obtained through the mapping relationship between the first feature vector and the text; the second text prediction probability corresponding to the first feature vector is obtained through the mapping relationship between the second feature vector and the text, and then the target text is determined by combining the first text prediction probability and the second text prediction probability.

[0172] As an example, the first feature vector and the second feature vector can also be fused to obtain a fused feature vector; and then the target text is identified according to the fused feature vector.

[0173] As an example, the speech recognition model further comprises a joint network configured to recognize target text corresponding to the speech of the target object from the mixed audio features according to the first feature vector and the second feature vector.

[0174] As an example, the embodiments of the present application process the mixed audio stream and recognize the speech of different objects respectively. For example, when the mixed audio stream includes speech of two persons, one object is determined as the target object, and the target text corresponding to the speech of the target object is recognized through the above S401-S404. Then, the second audio feature is used as the mixed audio feature, and the above S401-S404 is repeated to determine another object as the target object and recognize the target text corresponding to the speech of the another object. When the mixed audio stream includes speech of three or more persons, one object is determined as the target object, and the target text corresponding to the speech of the target object is recognized through the above S401-S404. Then, the second audio feature is updated based on the second audio feature, and the above S401-S404 is repeated to determine another object as the target object and recognize the target text corresponding to the speech of the another object. Then, the second audio feature is continuously updated based on the second audio feature, and the above S401-S404 is repeated to determine another object as the target object and recognize the target text corresponding to the speech of the another object, until the speech of all objects in the mixed audio stream is recognized respectively, and the text corresponding to different objects is generated. As shown in FIG. 4, X1 is the speech of object 1 in the mixed audio stream, Y1 is the text recognized from the speech of object 1; X2 is the speech of object 2 in the mixed audio stream, Y2 is the text recognized from the speech of object 2, and the text corresponding to each object is output respectively. Figure 9

[0175] ​In the embodiment of the present application, when the mixed audio stream is feature extracted, the mixed audio stream is feature extracted as first audio features and second audio features, the mixed audio stream is distinguished, and then divided into two processing paths, and the first audio features and the second audio features are processed respectively, wherein the first audio features are processed to obtain a first feature vector as a first path of feature processing; the second audio features are processed to obtain a second feature vector as a second path of feature processing, and the target text is finally recognized by fusing the first feature vector and the second feature vector. First, the mapping probability of the first audio features and the target text is calculated through the first path, and the mapping probability of the first audio features and the target text is predicted through the second path at the same time. The prediction results of the two paths are fused, which can improve the accuracy of speech recognition. Alternatively, it can be understood that in the same voice interaction scene, the speech of other objects in the mixed speech stream and the speech of the target object have a certain correlation relationship. The embodiment increases the second feature vector associated with the speech possibly emitted by the target object to predict the mapping probability of the speech of the target object and the target text, which can improve the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scene of the mixed audio stream, the speech recognition accuracy is high, and the speech recognition pause condition in the recognition process can be reduced.

[0176] Figure 6 An application flowchart of the speech recognition method provided by the embodiment of the present application is shown as an example in FIG. 1. Figure 6 As shown in FIG. 1, the speech recognition scenarios in the present example include but are not limited to voice control recognition, voice transcription, voice analysis, and audio / video recording. Different speech recognition scenarios correspond to different ways of outputting target texts. Based on this, in the actual application process, based on the selected speech recognition scenario, the first feature vector and the second feature vector are processed, and then the target text corresponding to the speech emitted by the target object in the mixed audio stream is recognized, and the target text is responded according to the recognition scenario. For example, if it is a voice control recognition scenario, the first feature vector and the second feature vector are identified based on the voice recognition scenario, and then the target text reflecting the control instruction is obtained, and the control instruction corresponding to the target text is executed.

[0177] In the example, the mixed audio stream is identified in combination with the specific scenario of speech recognition. In the recognition process, the probability of the text matching the corresponding attribute or feature is increased based on the attribute or feature of the speech recognition scenario, which can further improve the accuracy of speech recognition.

[0178] Figure 7 The flowchart of the speech recognition method provided by the embodiment of the present application is shown in FIG. 1. Figure TwoThe embodiment is based on the above embodiment. Since the streaming voice technology can process and feedback in real time, the efficiency and user experience of human-computer interaction are improved, which is a development trend of voice recognition. However, it also faces challenges in technical implementation and application scenarios. For example, in the scenario of voice recognition based on a model, the longer the input voice is, the greater the recognition error of the model will be. When the recognition error is large, the model cannot continue to recognize the voice, therefore, in the related technology, after recognizing audio of a certain length, a period of time needs to be paused before re-recognizing, for example, after recognizing a 30s audio stream, it will be paused, and the pause is frequent. Based on this, the embodiment of the present application reduces the number of pauses and prolongs the recognition length of the voice stream through the following embodiments. As shown in Figure 7 The recognition method comprises:

[0179] S701, performing feature extraction on the mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object;

[0180] S702, obtaining a first feature vector of the voice emitted by the target object according to the first audio features;

[0181] S703, predicting a second feature vector associated with the voice possibly emitted by the target object according to the second audio features;

[0182] In the embodiment, the specific implementation process of steps S501-S503 is similar to the implementation process of S401-S403 in the above Figure 4 embodiment, which will not be described here.

[0183] S704, predicting a backward probability of the voice possibly emitted by the target object at the next time step according to the first audio features;

[0184] In this step, the forward probability refers to the probability that the voice possibly emitted by the target object at the previous time step (t-1) corresponds to the first text at the current time step t. The backward probability refers to the probability that the voice possibly emitted by the target object at the next time step (t+1) corresponds to the second text according to the first audio features at the current time step t, wherein the second text can include multiple, and the backward probability of each second text is different. The first text, the target text and the second text are texts corresponding to the voice emitted by the target object in sequence, which constitute a continuous text corresponding to the voice stream of the target object.

[0185] The execution sequence between this step and S702 and S703 is not limited, which can be executed in sequence or simultaneously.

[0186] S705, predicting a target text corresponding to the voice emitted by the target object in the mixed audio stream according to the first feature vector, the second feature vector, the backward probability and the forward probability through a preset conditional probability.

[0187] As an example, according to the first feature vector, the second feature vector, the backward probability and the forward probability, the target text corresponding to the speech of the target object in the mixed audio stream is predicted by a preset conditional probability, specifically:

[0188] The first feature vector and the second feature vector are fused to obtain a fusion feature vector;

[0189] The backward probability of the speech possibly emitted by the target object at the next time step is predicted according to the fusion feature vector;

[0190] According to the forward probability predicted at the previous time step and the backward probability of the speech possibly emitted at the next time step, the target text corresponding to the speech of the target object in the mixed audio stream is predicted by a preset conditional probability.

[0191] As an example, the preset conditional probability is:

[0192] ;

[0193] wherein, is the forward probability, is the backward probability, and P(X) is a normalization constant; is the probability of predicting the target text based on the first feature vector and the second feature vector.

[0194] The target text corresponding to the speech is obtained by probability. In the above conditional probability, X is the speech, Y is the target text, is the probability of the speech corresponding to the text, if the probability of the speech corresponding to a text is the highest, the text is determined as the target text. For example, the probability of the speech corresponding to "hello" is 98%, and the target text corresponding to the speech is "hello".

[0195] In the model training process, the correlation between the conditional probability and the forward probability and the backward probability is established (such as the above conditional probability), and in the actual application process, after the fusion feature vector is obtained based on the fusion of the first feature vector and the second feature vector, the backward probability of the speech possibly emitted by the target object at the next time step is predicted by the fusion feature vector, and then the probability of the target text is predicted by the above conditional probability, that is, the probability of the speech emitted at the current time step and the target text is predicted by the forward probability predicted at the previous time step and the backward probability of the speech possibly emitted at the next time step. Based on the strong correlation between the streaming speech and the previous and subsequent speech, this prediction method can improve the prediction accuracy at the current time step and improve the recognition accuracy.

[0196] To illustrate the relationship between the forward probability, the backward probability and the prediction result of the current time, it is assumed that the mixed audio stream includes audio of time steps t1, t2, t3, …, tn. It is assumed that the target text predicted at the time step t1 is "big". When the audio of the time step t2 is recognized, the forward probability Q1 is obtained. According to the fusion feature vector based on the first audio feature and the second audio feature, the backward probability Q2 of the text corresponding to the speech that the target object is likely to make at the next time step (t3) is predicted, and then the probability of the target text based on Q1 and Q2 is predicted, which is P2. For example, the probability of "big" is the highest, and the target text corresponding to the fusion feature vector is likely to be "home". Through the feature vector, the probability of the text corresponding to the next speech of the target object is likely to be "good", and then the prediction probability of the target text "home" at the current time step t2 is improved based on the forward probability Q1 and the backward probability Q2. When the audio of the time step t3 is recognized, the P2 predicted at the time step t2 is the forward probability Q1 at the time step t3, and the backward probability Q2 of the text corresponding to the speech that the target object is likely to make at the next time step is predicted according to the fusion feature vector based on the first audio feature and the second audio feature, and then the probability of the target text based on Q1 and Q2 is obtained, which is P3. The P3 is the forward probability corresponding to the time step t4. By analogy, when the audio of the time step tn is recognized, the Pn-1 predicted at the time step t-1 is the forward probability Q1 at the time step tn, and the backward probability Q2 of the second text corresponding to the speech that the target object is likely to make at the next time step is predicted according to the fusion feature vector based on the first audio feature and the second audio feature, and then the probability of the target text based on Q1 and Q2 is obtained, which is Pn.

[0197] It should be noted that the forward probability Q1 is zero at the time step t1. The backward probability Q2 is zero at the time step tn, or based on the time sequence coding of the mixed audio stream, it is determined that there is no audio input after the time step tn, and then the forward probability of the second text corresponding to the speech that the target object is likely to make at the next time step at the time step tn is not needed to be predicted.

[0198] In this embodiment, first, the backward probability of the text corresponding to the speech that the target object is likely to make at the next time step is predicted based on the fusion feature vector of the first feature vector and the second feature vector, and then the forward probability and the backward probability are used to predict the target text, so that the prediction accuracy is improved. Since the backward probability of the text at the next time step is predicted at the current time step, the speech recognition accuracy is high, and the speech recognition model can continue to recognize, so that the pause frequency is reduced.

[0199] As an example, on the basis of all the above embodiments, after the target text corresponding to the speech made by the target object in the mixed audio stream is recognized, the method further includes:

[0200] When it is monitored that the target text corresponding to the voice of the target object at the current time is abnormal, the model parameters of the speech recognition model are optimized based on the abnormal scenario of the target text, the speech recognition application field, and the corrected text.

[0201] The corrected text is a correct text corresponding to the target text when the target text is abnormal.

[0202] The target text abnormality includes but is not limited to the following scenarios:

[0203] 1. In a voice control scenario, after the target text is recognized, the control instruction indicated by the target text is sent to the corresponding execution end, and the corresponding execution end does not feed back response information; or the execution end feeding back the response information does not match the execution end indicated by the voice (in this case, the user needs to feed back), which is the target text abnormality.

[0204] Taking the voice control scenario applied to a car as an example, assuming that the mixed audio stream includes voice of opening the air conditioner and closing the window, the speech recognition model recognizes that the target text corresponding to the mixed audio stream is opening the air conditioner and locking the door, and the car executes the actions of opening the air conditioner and locking the door. At this time, the user will manually or through voice feedback the relevant information based on the fact that the control executed by the car is different from the voice control instruction issued by the user, at which time it is monitored that the target text is abnormal.

[0205] 2. In a voice-to-text scenario, the target text is in an irregular font or has a large difference with the previous and subsequent texts, which is the target text abnormality.

[0206] 3. In a voice-to-text scenario, the target text does not match the speech recognition application field, for example, in a vehicle application scenario, medical terms appear in the text, which is the target text abnormality.

[0207] Further, the model parameters of the speech recognition model are optimized in combination with the abnormal type, the abnormal reason, and the like, so that the model parameters are optimized, and the recognition result is more accurate.

[0208] Further, the model parameters of the speech recognition model are optimized by the following manner:

[0209] The loss of the speech recognition model is adjusted according to the abnormal scenario of the target text, the speech recognition application field, and the corrected text. The corrected text can be fed back by the user or predicted. The corrected text is used as a label to train the speech recognition model, and the loss of the speech recognition model is adjusted. Optionally, the loss of the adjusted speech recognition model is smaller, which can improve the recognition accuracy of the speech recognition model.

[0210] Then, the forward probability algorithm model parameters and the backward probability algorithm model parameters are adjusted based on the adjusted loss.

[0211] Optionally, the relationship between the loss of the speech recognition model and the forward probability and the backward probability is:

[0212]

[0213] When Lrnnt(X, Y) is adjusted, the algorithm model parameters of the forward probability and the algorithm model parameters of the backward probability are adjusted.

[0214] According to the adjusted forward probability algorithm model parameters, the forward probability of the speech that the target object is likely to make at the previous moment of the current moment is predicted, and according to the adjusted backward probability algorithm model parameters, the backward probability of the speech that the target object is likely to make at the next moment of the current moment is predicted.

[0215] Optionally, the forward probability is predicted by the following way:

[0216]

[0217] Optionally, the backward probability is predicted by the following way:

[0218]

[0219] According to the forward probability, the backward probability, and the correction text, a gradient is calculated.

[0220] Optionally, the gradient is calculated by the following way:

[0221]

[0222] Further, based on the gradient, the model parameters of the speech recognition model are adjusted.

[0223] By the above method, the model parameters of the speech recognition model are quickly optimized.

[0224] Optionally, in the embodiment, as shown in Figure 6 , the model parameters of the speech recognition model are optimized through the cloud. Optimizing the model parameters of the speech recognition model in the cloud can realize real-time OTA based on the cloud and the end, deploy in the cloud, reduce the computing power of the end, and avoid affecting the normal use of other functions of the end when optimizing the model parameters.

[0225] It can be understood that the speech recognition model in the present application is set based on a heuristic error distribution method, and heuristic rules are used to quickly select optimal model parameters. Specifically, during model training, a basic model is pre-trained through limited labeled data. Compared with conventional model training relying on perfect labeled data, the present application can reduce data acquisition costs and reduce training difficulty.​​​​

[0226] Then the base model is applied to the end. By using the model in the end, based on the identification result of the model, whether the current solution of the model is optimal is measured.

[0227] By applying the statistical analysis of the application field, the error mode, etc., according to the error type, the reason, the scene, etc., based on the heuristic rule, the new model parameter is obtained, if the model parameter makes the recognition result of the speech recognition model more accurate, that is, the current solution of the model is more optimal, the speech recognition model is updated, in the continuous process, the model is gradually improved based on the heuristic error training method, and the same error is learned and reduced, and the accuracy of the speech recognition model is gradually improved. In the same application field, through continuous learning, to a certain extent, non-stop streaming speech recognition can be realized.

[0228] In the embodiment, the way in which the heuristic rule obtains the new model parameter can refer to the way listed above, that is, the gradient is calculated through the forward probability and the backward probability, and then the modified direction of the model parameter is reflected based on the gradient, and then the revised model parameter is obtained.

[0229] In the embodiment, when the recognition of the target text is abnormal, the model parameter of the speech recognition model is optimized through the abnormal scene of the target text, the speech recognition application field and the correction text, and the recognition accuracy of the speech recognition model is improved. It should be noted that in the application process of the embodiment, based on the recognition exception, that is, how to obtain the minimum loss of the speech recognition model under the abnormal situation is learned, and then the speech recognition model is continuously corrected, rather than re-identifying the speech after the gradient explosion of the speech recognition model. Therefore, compared with the related art, based on the continuous optimization and correction of the speech recognition model, the accuracy of the speech recognition model can be higher and higher, and then the indefinite length streaming speech can be recognized.

[0230] In addition, if the speech recognition model is migrated to a new application field, the heuristic error distribution rule can also be used to gradually adjust the model to adapt to the characteristics of the new field, so that the speech recognition model is easy to adapt to various fields.

[0231] The speech recognition model mentioned in the above embodiments is obtained after being trained by the training method of the speech recognition model provided in the embodiment, and the speech recognition model is specifically referred to in Figure 8 In the embodiment, the training method comprises:

[0232] S801, obtaining a training sample, the training sample comprising a sample mixed audio stream and a sample mixed audio stream corresponding sample label;

[0233] In this step, the sample label is the actual target text (hereinafter referred to as a sample label), which can be a text annotation of a sample mixed audio stream or a text set according to the mixed sample audio stream.

[0234] In S802, the sample mixed audio stream is input into the speech recognition model, and the following processing is performed in the speech recognition model to identify the target sample label corresponding to the speech emitted by the target object in the sample mixed audio stream: feature extraction is performed on the sample mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object; a first feature vector of the speech emitted by the target object is obtained according to the first audio features, and a second feature vector associated with the speech possibly emitted by the target object is predicted according to the second audio features; and the target sample label is identified according to the first feature vector and the second feature vector.

[0235] In the embodiments of the present application, the speech recognition model includes a mask network, a long short-term memory network, a prediction network, and a joint network. The mask network is used to extract the first audio features and the second audio features of different persons by performing feature extraction on the sample mixed audio stream. The long short-term memory network and the prediction network form two prediction paths, wherein the long short-term memory network is a first path (main path) for predicting the corresponding first feature vector based on the first audio features of the target object, and the prediction network is a second path (auxiliary path) for predicting the second feature vector of the speech possibly emitted by the target object based on the second audio features. That is, the long short-term memory network and the prediction network form a double prediction path, so that the speech recognition model is a model formed by double path modeling. After obtaining the first feature vector and the second feature vector based on the double prediction path, the joint network is used to fuse and predict the first feature vector and the second feature vector, and then obtain the probability of the target text.

[0236] Based on the structure of the speech recognition model, after the sample mixed audio stream is input into the speech recognition model, the sample mixed audio stream is first input into the mask network and is processed by the mask network. Specifically, in some embodiments, the mask network includes a mask encoder, the sample mixed audio stream is input into the mask encoder, the sample mixed audio stream is processed based on the mask encoder to predict a mask of the speech emitted by the target object; and the sample mixed audio stream is processed based on the mask and a complement of the mask corresponding to the mask to obtain the first audio features of the target object and the second audio features of other objects except the target object, so as to distinguish the audio features of different objects.

[0237] Or, in some embodiments, the mask network includes a preprocessing encoder and a mask encoder, the sample mixed audio stream is input into the preprocessing encoder, the sample mixed audio stream is preprocessed based on the preprocessing encoder to obtain sample mixed audio features. Then the sample mixed audio features are input into the mask encoder, the sample mixed audio features are processed based on the mask encoder to predict the mask of the voice emitted by the target object; according to the mask, the sample mixed audio features and the complement of the mask, the sample mixed audio features are processed to obtain the first audio features of the target object and the second audio features of other objects except the target object, so as to distinguish the audio features of different objects.

[0238] As an example, based on the mask and the sample mixed audio features, the first audio features of the target object are obtained; based on the complement and the sample mixed audio features, the second audio features are obtained.

[0239] After the sample mixed audio stream passes through the mask network, the feature extraction of the sample mixed audio stream is completed, and the feature extraction is the first audio features and the second audio features. Then the first audio features are input into the long short-term memory network, the first audio features are feature extracted based on the long short-term memory network to obtain the first feature vector of the voice emitted by the target object. The second audio features are input into the prediction network, the second audio features are predicted based on the prediction network to generate the corresponding second feature vector. By processing the first audio features and the second audio features through the double paths respectively, the audio features of a single target object in the mixed audio stream can be separated to obtain the first feature vector, and the second feature vector corresponding to the voice emitted by the target object can be predicted based on the second audio features of other objects in the mixed audio stream. The second feature vector improves the probability between the first feature vector and the target text, and further improves the recognition accuracy.

[0240] The first audio features pass through the long short-term memory network to obtain the first feature vector, the second audio features pass through the prediction network to obtain the second feature vector, and then the first feature vector and the second feature vector are input into the joint network respectively, the first feature vector and the second feature vector are fused based on the joint network to obtain a fusion feature vector; according to the fusion feature vector, the target text corresponding to the voice emitted by the target object in the sample mixed audio stream is recognized.

[0241] As an example, before the first audio feature is input into the Long Short-Term Memory (LSTM) network, it needs to be encoded along with the sample labels to facilitate the time-synchronized mapping of audio features and sample labels. Therefore, the speech recognition model also includes an audio encoder and a label encoder. The audio encoder encodes the first audio feature, and the label encoder encodes the sample labels (i.e., the label samples). Specifically, the audio encoder encodes the first audio feature based on time sequence, and the label encoder also encodes the sample labels based on time sequence, thus obtaining time-synchronized first audio features and sample labels. For example, the audio features corresponding to the mixed audio stream are X = {x1, x2, ..., x...}. t The corresponding sample label is Y = {y1, y2, ..., y}. u}, where t is the temporal length of the mixed audio stream and u is the length of the sample label.

[0242] In this embodiment, during the processing of the mixed audio stream of samples by the speech recognition model, the sample labels are synchronously converted based on time series. During time synchronization, the original time series data is converted into a feature vector representation. The Long Short-Term Memory (LSTM) network then identifies, classifies, and processes the data in real time, capturing time dependencies and patterns of change. This continuous and dynamic processing and recognition of time series data improves the accuracy and reliability of predictions. It also provides the possibility of deploying the speech recognition model on the edge.

[0243] S803, calculates the loss value of the target text relative to the corresponding sample label based on the total loss function;

[0244] S804 adjusts the model parameters of the speech recognition model based on the loss value.

[0245] It should be noted that, in this embodiment, the mapping relationship between the target text and the mixed audio stream is represented by the association probability. For example, if the association probability between the speech emitted by the target object at the current time step and the text "you" in the mixed audio stream is A%, which is the maximum probability, then the text "you" is determined to be the target text corresponding to the speech emitted by the target object at the current time step. This embodiment calculates the loss value of the target text relative to the corresponding sample label using a total loss function, and then continuously adjusts the magnitude of the association probability based on the loss function, resulting in high recognition accuracy of the speech recognition model.

[0246] As an example, a total loss function is constructed based on the error between the target text and the target sample label predicted by the first audio feature and the error between the target text and the target sample label predicted by the second audio feature.

[0247] Optionally, the total loss function is:

[0248] ;

[0249] wherein, X is a mixed audio stream, H1 is a first audio feature, H2 is a second audio feature, Y1 is a first sample label corresponding to the first audio feature, Y2 is a second sample label corresponding to the second audio feature; Lrnnt is a loss function between the input audio feature H and the corresponding sample label Y; L heat is a loss value.

[0250] It should be noted that the total loss function reflects a heuristic error distribution method, which is a heuristic error distribution (selecting a label distribution based on heuristic information), and the model is trained to minimize L heat , the model learns to match the feature representations H1 and H2 with their corresponding label sequences (target text sequences) Y1 and Y2, respectively. The mixed audio stream X is processed based on the mask to obtain H1 and H2, and the error between the predicted target text of H1 and the first sample label Y1, and the error between the predicted target text of H2 and the second sample label Y2 are calculated by the probability condition, and the total loss function reflects the sum of the errors.

[0251] When the loss value is greater than or equal to a preset threshold, the model parameters of the speech recognition model are adjusted, so that when learning the next training sample, the loss value can be reduced, and thus the recognition accuracy of the speech recognition model is improved.

[0252] In the embodiments of the present application, by setting the speech recognition model to extract the features of the mixed audio stream as the first audio feature and the second audio feature when extracting features, the mixed audio stream is distinguished and then divided into two processing paths, and the first audio feature and the second audio feature are processed respectively, wherein the first audio feature is processed to obtain a first feature vector as a first path of feature processing; the second audio feature is processed to obtain a second feature vector as a second path of feature processing, and the target text is finally recognized by fusing the first feature vector and the second feature vector. Based on learning the association between the first audio feature and the text, the association between the second audio feature and the text is also learned, and the probability of learning the relationship between the mixed audio stream and the text is increased through the two paths, which can improve the recognition accuracy of each target object in the mixed audio stream.

[0253] In further embodiments, the training method of the speech recognition model further comprises: predicting the backward probability that the speech possibly emitted by the target object at the next time step corresponds to the second sample label according to the first audio feature;

[0254] Based on the first feature vector, the second feature vector, the backward probability, and the forward probability that the speech emitted by the target object at the previous time step corresponds to the first sample label, the target text is predicted by a preset conditional probability.

[0255] That is, in this embodiment, the first feature vector, the second feature vector, the backward probability and the forward probability are combined to comprehensively identify the target text corresponding to the voice emitted by the target object in the sample mixed audio stream, so as to improve the accuracy of voice recognition.

[0256] Optionally, the preset conditional probability is:

[0257]

[0258] wherein, is the forward probability, is the backward probability, P(X) is a normalization constant, is the probability of predicting the target text based on the first feature vector and the second feature vector.

[0259] In this example, the RNNT model of the audio encoder, the label encoder, the long short-term memory network (LSTM) and the joint network models the conditional probability relationship between the audio features (acoustic features) and the sample labels (target text), directly maps the acoustic feature sequence to the label sequence (target text sequence) and synchronously converts, and the training and decoding processes of the model both depend on the calculation of the conditional probability.

[0260] As an example, the voice recognition model further includes an audio encoder and a label encoder, and the training method further includes:

[0261] Obtaining a training sample, the training sample including a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream;

[0262] In this step, the sample label is actually the target text (hereinafter referred to as the sample label), which can be a text annotation of the sample mixed audio stream or a text set according to the mixed sample audio stream.

[0263] Inputting the sample mixed audio stream into the voice recognition model, and performing the following processing in the voice recognition model to identify the target text corresponding to the voice emitted by the target object in the sample mixed audio stream: performing feature extraction on the sample mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object; obtaining a first feature vector of the voice emitted by the target object according to the first audio features, and predicting a second feature vector associated with the voice possibly emitted by the target object according to the second audio features; and identifying the target text according to the first feature vector and the second feature vector.

[0264] ​Based on the above, in the embodiments of the present application, the training process of the speech recognition model can be: the system is powered on, the MIC receives the mixed acoustic sequence in the cabin, and the pre-processing encoder (MixEnc) extracts the features. The mask encoder (MaskEnc) estimates a mask M. Through the mask M and the pre-processed features, the feature representations H1 and H2 of the two speakers of the overlapping audio can be calculated. The overlapping audio speech acoustic feature sequence X={x1,x2..., x t} and its corresponding label sequence (target text sequence) Y={y1,y2..., y u} are set, where t is the length of the acoustic sequence, and u is the length of the label sequence (target text sequence). The label encoder converts the label sequence (target text sequence) Y into an embedded representation. The dual-path modeling captures the local features and global context information of the speech signal through the sound features and the time dependence between the voices of the two speakers in the single-person audio and multi-person audio environment, and in different signal-to-noise ratio conditions. The audio encoder and the label encoder network calculate the forward probability and the backward probability of each time step. The probability of all possible label sequence (target text sequence) paths (including blank labels) is calculated. The normalized probability distribution is calculated by the forward and backward probabilities. The RNNT model of the audio encoder, the label encoder, the long short-term memory network (LSTM), and the joint network models the conditional probability relationship between the acoustic features and the label sequence (target text sequence) in a time-synchronous sequence conversion. The acoustic feature sequence is mapped to the label sequence (target text sequence) and the audio features are converted into text simultaneously, and the corresponding label sequence (target text sequence) is predicted.

[0265] As an example, the speech recognition model also performs the following processing: inputting the first audio feature into the audio encoder, encoding the first audio feature based on the audio encoder, encoding the sample label according to the label encoder, predicting the backward probability of the second sample label corresponding to the speech that the target object may utter at the next time step according to the first audio feature; predicting the gradient of the model parameter corresponding to the loss function according to the backward probability and the forward probability of the first sample label corresponding to the speech uttered by the target object predicted at the previous time step;

[0266] As an example, the gradient is calculated by the following formula:

[0267] ;

[0268] wherein, and are the model parameters; Lrnnt is the loss function between the input audio feature H and the corresponding sample label Y; is the forward probability, is the backward probability.

[0269] The loss value of the target text with respect to the corresponding sample label is calculated based on the total loss function; the order of this step and the gradient prediction step described above is not limited, and they can be executed simultaneously or sequentially.

[0270] When the loss value is greater than or equal to a preset threshold, the model parameters of the speech recognition model are adjusted according to the gradient.

[0271] In this example, the gradient reflects the sensitivity of the difference (loss value) between the target text recognized by the speech recognition model and the sample label to the change of the model parameters (partial derivative of the model parameters), which is used to guide how to adjust the model parameters to minimize the total loss function. By calculating the gradient and then adjusting the model parameters based on the gradient, on the one hand, the difference between the target text recognized by the speech recognition model and the sample label can be kept within a small range, so that the speech recognition model can continuously process indefinite-length sequences (streaming speech) and avoid the need to pause for 30 seconds when the recognized probability is low. On the other hand, the speech recognition model can quickly learn the association between the mixed audio stream and the sample label, reducing the learning samples and improving the learning efficiency.

[0272] In this embodiment, the forward probability represents the probability of observing the sample audio feature and the sample label before time step t and sample label u, and is defined as the forward probability. Optionally, the forward probability is predicted by the following formula:

[0273] ;

[0274] The backward probability represents the probability of observing the sample audio feature and the sample label after time step t and sample label u, and is defined as the backward probability. Optionally, the backward probability is predicted by the following formula:

[0275] ;

[0276] In this embodiment, the total loss function is:

[0277] ;

[0278] where X is the mixed audio stream, H1 is the first audio feature, H2 is the second audio feature, Y1 is the first sample label corresponding to the first audio feature, and Y2 is the second sample label corresponding to the second audio feature; Lrnnt is the RNNT loss function, which is the loss function between the input audio feature H and the corresponding sample label Y; Lheat is a loss value.

[0279] In this example, the total loss function is composed of Lrnnt(Y1, H1) + Lrnnt(Y2, H2), where H1 corresponds to the first audio features extracted from the sample mixed audio stream, H2 corresponds to the second audio features extracted from the sample audio stream, Y1 is the sample label identified based on the first audio features, Y2 is the sample label identified based on the second audio features, Lrnnt(Y1, H1) represents the loss value of the probability of identifying the first sample label based on the first audio features, and Lrnnt(Y2, H2) represents the loss value of the probability of identifying the second sample label based on the second audio features. For the mixed audio stream, the total loss value is calculated based on the loss values of different objects, which can ensure that the speech recognition model balances the learning efficiency of different objects during the learning process, so that the recognition accuracy of each object is balanced. On the other hand, the model is adjusted more carefully during the training process to reduce specific types of errors, which is good in scalability and efficiency, avoids considering all possible label arrangements, and thus reduces the calculation and memory requirements.

[0280] Optionally, Lrnnt can be calculated by the following formula:

[0281] ;

[0282] wherein, is a forward probability, is a backward probability, U is the length of the sample label, and u is the sample label corresponding to the current time step.

[0283] In this example, the loss between the input audio features and the sample label is calculated by combining the forward probability and the backward probability, which can make the loss tend to update during the learning process of the speech recognition model. In this way, the speech recognition model learned has smaller recognition errors during the application process, and can thus learn longer audio streams.

[0284] Based on the embodiments of the present application, the speech recognition model trained can realize timely recognition and reaction of mixed audio streams, avoiding the situation that a pause is needed after recognizing a certain length of audio stream.

[0285] It should be noted that in the embodiments of the present application, a basic model is pre-trained through limited sample labels (allowing errors) to reduce training samples and reduce data acquisition costs. The computational power requirement of the basic model is small, which facilitates deployment to the end side, such as application to a car, and application of the speech recognition model to the electronic devices of the car.

[0286] In the use process of the speech recognition model, the model parameters of the speech recognition model are updated through the heuristic error distribution rule. For details of the updating process, refer to the updating process described in the above embodiments. The model parameters of the speech recognition model are updated based on specific abnormal conditions in the use process, so that the model is more suitable for speech recognition in the application scenario, and the recognition accuracy is higher and higher.

[0287] The embodiment of the present application also provides a speech recognition device, comprising:

[0288] The extraction module is configured to perform feature extraction on the mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object.

[0289] The first prediction module is configured to obtain a first feature vector of the speech emitted by the target object according to the first audio features.

[0290] The second prediction module is configured to predict a second feature vector of the speech possibly emitted by the target object according to the second audio features.

[0291] The recognition processing module is configured to recognize target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

[0292] The speech recognition device provided by the embodiment can perform the method provided by the above method embodiment, and has similar implementation principles and technical effects. Therefore, the embodiment will not be described here.

[0293] The embodiment of the present application also provides a speech recognition model training device, comprising:

[0294] The acquisition module is configured to acquire training samples, wherein the training samples comprise sample mixed audio streams and sample labels corresponding to the sample mixed audio streams.

[0295] The speech recognition model is configured to perform the following processing on the input sample mixed audio stream to recognize target text corresponding to the speech emitted by the target object in the sample mixed audio stream: performing feature extraction on the sample mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object; obtaining a first feature vector of the speech emitted by the target object according to the first audio features; predicting a second feature vector of the speech possibly emitted by the target object according to the second audio features; and recognizing the target text according to the first feature vector and the second feature vector.

[0296] The training module is configured to calculate a loss value of the target text relative to the corresponding sample label based on a total loss function, and adjust the model parameters of the speech recognition model according to the loss value.

[0297] The training device of the speech recognition model provided in the embodiment can execute the method provided in the method embodiment, and has similar implementation principles and technical effects, which will not be described here again.

[0298] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor.

[0299] The memory is used for storing computer programs / instructions, and the processor is used for executing the computer programs / instructions stored in the memory to realize the method related to each of the above embodiments.

[0300] The electronic device further comprises a communication interface and a CAN bus, wherein the processor is used for providing computing power and control capability, and can be a GPU, a CPU, an NPU, an MCU, an FPGA, etc. The storage device comprises an internal memory and a non-volatile memory. The non-volatile memory stores a computer program for realizing the above method. The internal memory provides an environment for program starting and running. The communication interface is used for wired or wireless communication with an external terminal.

[0301] In the embodiment, the electronic device can be an electronic component in a car or the car itself.

[0302] The present application also provides a computer readable storage medium / computer program product, wherein the computer readable storage medium stores computer execution instructions, and the computer program product comprises computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used for realizing the method related to each of the above embodiments.

[0303] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent replacement or transformation of the present application made by those skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. A voice recognition method, characterized by, The method comprises: performing feature extraction on a mixed audio stream to obtain first audio features of a target object and second audio features of other objects except the target object, the mixed audio stream comprising speech of at least two objects, the target object being one object distinguished from the at least two objects; obtaining, according to the first audio features, a first feature vector of speech emitted by the target object; predicting, according to the second audio features, a second feature vector associated with speech possibly emitted by the target object; recognizing, according to the first feature vector and the second feature vector, target text corresponding to the speech emitted by the target object in the mixed audio stream.

2. The identification method according to claim 1, characterized in that, The recognizing, according to the first feature vector and the second feature vector, target text corresponding to the speech emitted by the target object in the mixed audio stream, comprises: performing time synchronization fusion on the first feature vector and the second feature vector to obtain a fused feature vector; recognizing, according to the fused feature vector, target text corresponding to the speech emitted by the target object in the mixed audio stream.

3. The identification method according to claim 1, characterized in that, The recognition method further comprises: predicting, according to the first audio features, a backward probability of speech possibly emitted by the target object at a next time step; The recognizing, according to the first feature vector and the second feature vector, target text corresponding to the speech emitted by the target object in the mixed audio stream, comprises: based on the first feature vector, the second feature vector, the backward probability, and a forward probability of the speech emitted by the target object predicted at a previous time step, predicting, by a preset conditional probability, target text corresponding to the speech emitted by the target object in the mixed audio stream.

4. The identification method of claim 1, wherein, The feature extraction on the mixed audio stream to obtain the first audio features of the target object and the second audio features of other objects except the target object comprises: performing preprocessing on the mixed audio stream to obtain mixed audio features; predicting, according to the mixed audio features, a mask of speech emitted by the target object; obtaining, according to the mask and the mixed audio features, the first audio features of the target object; obtaining, according to a complement of the mask and the mixed audio features, the second audio features of other objects except the target object.

5. The identification method according to any one of claims 1 to 4, characterized in that, The method is applied to a computing device deployed with a speech recognition model, the speech recognition model comprising a mask network, a long short-term memory network, a prediction network, and a joint network, wherein: the mask network is configured to perform feature extraction on the mixed audio stream to obtain first audio features of a target object and second audio features of other objects except the target object; the long short-term memory network is configured to obtain, according to the first audio features, a first feature vector of speech emitted by the target object; the prediction network is configured to predict, according to the second audio features, a second feature vector associated with speech possibly emitted by the target object; the joint network is configured to recognize, according to the first feature vector and the second feature vector, target text corresponding to the speech emitted by the target object in the mixed audio stream.

6. The identification method according to claim 5, characterized in that, The recognition obtains the target text corresponding to the voice emitted by the target object in the mixed audio stream, and further comprises: When it is monitored that the target text corresponding to the voice emitted by the target object is abnormal at the current time, the model parameters of the speech recognition model are optimized based on the abnormal scene of the target text, the application field of speech recognition and the corrected text, wherein the corrected text is the correct text corresponding to the correction when the target text is abnormal.

7. The identification method according to claim 6, characterized in that, The model parameters of the speech recognition model are optimized based on the abnormal scene of the target text, the application field of speech recognition and the corrected text, comprising: According to the abnormal scene of the target text, the application field of speech recognition and the corrected text, the speech recognition model is trained, and the loss of the speech recognition model is adjusted; The forward probability algorithm model parameters and the backward probability algorithm model parameters are adjusted based on the adjusted loss; According to the adjusted forward probability algorithm model parameters, the forward probability of the voice that the target object may emit at the previous moment of the current moment is predicted, and based on the adjusted backward probability algorithm model parameters, the backward probability of the voice that the target object may emit at the next moment of the current moment is predicted; The gradient is calculated according to the forward probability, the backward probability and the corrected text; The model parameters of the speech recognition model are adjusted based on the gradient.

8. The identification method according to claim 6 or 7, characterized in that, The model parameters of the speech recognition model are optimized through the cloud. 9.A method for training a speech recognition model, the method comprising: Comprising: Obtaining training samples, the training samples comprising sample mixed audio streams and sample labels corresponding to the sample mixed audio streams; The sample mixed audio stream is input into the speech recognition model, and the following processing is performed in the speech recognition model to identify the target text corresponding to the voice emitted by the target object in the sample mixed audio stream: performing feature extraction on the sample mixed audio stream to obtain first audio features of the target object and second audio features of other objects except the target object; obtaining a first feature vector of the voice emitted by the target object according to the first audio features, and predicting a second feature vector associated with the voice that the target object may emit according to the second audio features; and identifying the target text according to the first feature vector and the second feature vector; The loss value of the target text relative to the corresponding target sample label is calculated based on the total loss function; The model parameters of the speech recognition model are adjusted according to the loss value.

10. The training method of claim 9, wherein, The speech recognition model comprises a mask network, and the feature extraction on the sample mixed audio stream to obtain the first audio features of the target object and the second audio features of other objects except the target object comprises: The sample mixed audio stream is input into the preprocessing encoder of the mask network, the sample mixed audio features are obtained based on the preprocessing of the sample mixed audio stream by the preprocessing encoder; The sample mixed audio features are input into the mask encoder of the mask network, and the mask of the voice emitted by the target object is predicted based on the processing of the sample mixed audio features by the mask encoder; The first audio features of the target object are obtained according to the mask and the sample mixed audio features; According to the complement corresponding to the mask and the sample mixed audio feature, a second audio feature of an object other than the target object is obtained.

11. The training method of claim 9, wherein, The speech recognition model comprises a long short-term memory network, and the first feature vector of the speech emitted by the target object is obtained according to the first audio feature, comprising: The first audio feature is input into the long short-term memory network, the long short-term memory network is used for feature extraction on the first audio feature, and the first feature vector of the speech emitted by the target object is obtained; And / or, the speech recognition model comprises a prediction network, and the second feature vector of the speech possibly emitted by the target object is predicted according to the second audio feature, comprising: The second audio feature is input into the prediction network, the prediction network is used for predicting the speech possibly emitted by the target object corresponding to the second audio feature, and the corresponding second feature vector is generated.

12. The training method of claim 9, wherein, The training method further comprises: The backward probability of the speech possibly emitted by the target object at the next time step is predicted according to the first audio feature; The target text is identified according to the first feature vector and the second feature vector, comprising: The target text is predicted by a preset conditional probability based on the first feature vector, the second feature vector, the backward probability and the forward probability of the speech emitted by the target object predicted at the previous time step.

13. Training method according to any one of claims 9 to 12, characterized in that, The speech recognition model further comprises an audio encoder and a label encoder, and the training method further comprises: The first audio feature is input into the audio encoder, the audio encoder is used for encoding the first audio feature, and the sample label is encoded according to the label encoder; The backward probability of the speech possibly emitted by the target object at the next time step is predicted according to the first audio feature; The gradient of the loss function to the model parameter is predicted according to the backward probability and the forward probability of the speech emitted by the target object predicted at the previous time step. The model parameter of the speech recognition model is adjusted according to the loss value, comprising: When the loss value is greater than or equal to a preset threshold, the model parameter is adjusted according to the gradient.

14. The training method of claim 13, wherein, The training method further comprises: The total loss function is constructed based on the error of the target text and the target sample label predicted according to the first audio feature, and the error of the target text and the target sample label predicted according to the second audio feature.

15. A speech recognition apparatus, characterized by comprising: Comprise: The extraction module is used for feature extraction on the mixed audio stream, to obtain the first audio feature of the target object and the second audio feature of the object other than the target object; The first prediction module is used for obtaining the first feature vector of the speech emitted by the target object according to the first audio feature; The second prediction module is used for predicting the second feature vector associated with the speech possibly emitted by the target object according to the second audio feature; The identification processing module is used for identifying the target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector. 16.A device for training a speech recognition model, comprising: Comprise: An acquisition module acquires a training sample, the training sample including a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream; A speech recognition model is configured to perform the following processing on an input sample mixed audio stream to identify a target text corresponding to speech emitted by a target object in the sample mixed audio stream; performing feature extraction on the sample mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects other than the target object; obtaining a first feature vector of speech emitted by the target object according to the first audio feature, and predicting a second feature vector associated with speech possibly emitted by the target object according to the second audio feature; and identifying the target text according to the first feature vector and the second feature vector; A training module is configured to calculate a loss value of the target text relative to the corresponding sample label based on a total loss function, and adjust model parameters of the speech recognition model according to the loss value.

17. An electronic device, comprising: The electronic device includes a memory and a processor. The memory is configured to store computer programs / instructions, and the processor is configured to implement the method according to any one of claims 1 to 14 based on the computer programs / instructions stored in the memory.

18. A vehicle characterized by comprising: The electronic device includes a memory and a processor. The memory is configured to store computer programs / instructions. The processor is configured to implement the method according to any one of claims 1 to 8 based on the computer programs / instructions stored in the memory. The computer readable storage medium stores computer programs / instructions, or the computer program product includes computer programs / instructions, and the computer programs / instructions are executed by the processor to implement the method according to any one of claims 1 to 14.

19. A computer readable storage medium / computer program product, characterized in that, ​

Images