Speech recognition method and device, model training method and device, electronic equipment and vehicle

Abstract

The invention relates to a speech recognition method, a model training method and device, electronic equipment and a vehicle, and the method comprises the steps: carrying out the feature extraction of a mixed audio stream, obtaining a first audio feature of a target object and a second audio feature of other objects except the target object, and enabling the mixed audio stream to comprise the speech of at least two objects, the target object is an object distinguished from at least two objects; according to the first audio feature, obtaining a first feature vector of voice emitted by the target object; according to the second audio feature, predicting to obtain a second feature vector associated with the voice possibly sent by the target object; and according to the first feature vector and the second feature vector, identifying to obtain a target text corresponding to the voice emitted by the target object in the mixed audio stream. According to the embodiment of the invention, the voice of the target object in the mixed audio stream is identified through the features extracted by the double paths, so that the identification accuracy of the mixed audio stream can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method, a model training method, a device, an electronic device and a vehicle. Background Art

[0002] Speech recognition refers to the human-computer interaction process of converting human language into text or commands. Currently, speech recognition technology is widely used in scenarios such as voice interaction, speech transcription, speech analysis, audio and video recording, etc.

[0003] In the related art, the recognition of speech is carried out by using a deep learning model to train a large amount of data, learn the relationship between the audio signal and the corresponding text, and thus can convert the user's speech into text. In the human-computer interaction scenario, the speech collected is generally mixed speech (the sound of multiple people speaking or environmental noise), so the recognition scenario of mixed speech stream is more extensive. The deep learning model in the related art processes the mixed speech stream by continuously extracting the speech features emitted by a single object from the mixed speech stream, and then identifying the target text corresponding to the speech features. This speech recognition method has low accuracy. For the recognition of mixed speech streams, the model needs to recognize continuously. As the length of the speech stream increases, the recognition accuracy gradually decreases. Therefore, after recognizing a certain length of mixed speech stream, the model needs to pause for a period of time and then restart the recognition. Therefore, the speech recognition method in the related art has low accuracy and requires frequent pauses.

[0004] Therefore, there is an urgent need to provide a speech recognition method with higher recognition accuracy to reduce pauses. Summary of the invention

[0005] One of the objects of the present invention is to provide a speech recognition method to solve the problem of low accuracy of mixed speech recognition in the prior art; the second object is to provide a method for training a speech recognition model; the third object is to provide a speech recognition device; the fourth object is to provide a training device for a speech recognition model; the fifth object is to provide an electronic device; and the sixth object is to provide a vehicle.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A speech recognition method, the method comprising:

[0008] Extracting features from a mixed audio stream to obtain a first audio feature of a target object and a second audio feature of another object other than the target object, wherein the mixed audio stream includes speech of at least two objects, and the target object is an object distinguished from the at least two objects;

[0009] Obtaining a first feature vector of the speech emitted by the target object according to the first audio feature;

[0010] Predicting, based on the second audio feature, a second feature vector associated with the speech that may be uttered by the target object;

[0011] A target text corresponding to the speech uttered by the target object in the mixed audio stream is identified based on the first feature vector and the second feature vector.

[0012] According to the above technical means, when extracting features from a mixed audio stream, the features of the mixed audio stream are extracted as a first audio feature and a second audio feature, so as to distinguish the mixed audio stream, and then the mixed audio stream is divided into two processing paths, and the first audio feature and the second audio feature are processed respectively, wherein the first audio feature is processed to obtain a first feature vector as the first path for feature processing; the second audio feature is processed to obtain a second feature vector as the second path for feature processing, and the target text is finally recognized by fusing the first feature vector and the second feature vector. First, the mapping probability of the first audio feature and the target text is calculated through the first path, and the mapping probability of the first audio feature and the target text is predicted through the second path. The prediction results of the two paths are fused, which can improve the accuracy of speech recognition. Or it can be understood that in the same voice interaction scenario, the voices of other objects in the mixed voice stream have a certain correlation with the voice of the target object. This embodiment predicts the mapping probability of the target object's voice and the target text by adding a second feature vector associated with the voice that may be uttered by the target object, thereby improving the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scenario of the mixed audio stream, the voice recognition accuracy is high and the pauses in voice recognition during the recognition process can be reduced.

[0013] Further, the identifying and obtaining a target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector includes:

[0014] Performing time-synchronous fusion on the first feature vector and the second feature vector to obtain a fused feature vector;

[0015] According to the fused feature vector, a target text corresponding to the speech uttered by the target object in the mixed audio stream is identified.

[0016] According to the above technical means, by converting the first audio feature into a first feature vector, converting the second audio feature into a second feature vector, and then fusing them, based on the post-fusion method, the accuracy can be higher and the amount of calculation can be reduced, providing the possibility for the recognition method to be deployed on the terminal side.

[0017] Furthermore, the identification method further includes:

[0018] Predicting a backward probability of a speech that the target object may utter at a next time step based on the first audio feature;

[0019] The step of identifying a target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector includes:

[0020] Based on the first feature vector, the second feature vector, the backward probability and the forward probability of the speech emitted by the target object predicted in the previous time step, the target text corresponding to the speech emitted by the target object in the mixed audio stream is obtained through preset conditional probability prediction.

[0021] According to the above technical means, the forward probability and the backward probability are used to associate the front and back features of the speech sequence. For the recognition of streaming audio features, the front and back features are used to form the features of words or sentences. The target text corresponding to the speech emitted by the target object is predicted based on the forward probability and the backward probability. This can further improve the recognition accuracy and reduce the frequency of pauses of the speech recognition model when the recognition accuracy is reduced.

[0022] Further, the extracting features of the mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects other than the target object includes:

[0023] Preprocessing the mixed audio stream to obtain mixed audio features;

[0024] Predicting a mask of speech emitted by a target object based on the mixed audio features;

[0025] Obtaining a first audio feature of the target object according to the mask and the mixed audio feature;

[0026] According to the complement corresponding to the mask and the mixed audio feature, a second audio feature of an object other than the target object is obtained.

[0027] According to the above technical means, the mixed audio stream is preprocessed to extract mixed audio features that conform to the format for easy mask processing. By performing mask processing on the mixed audio stream, the features corresponding to the voices of different objects in the mixed audio stream are distinguished, and the mixed audio stream is separated into at least two paths corresponding to the required audio features.

[0028] Further, the invention is applied to a computing device having a speech recognition model deployed thereon, wherein the speech recognition model includes a mask network, a long short-term memory network, a prediction network, and a joint network, wherein:

[0029] The mask network is used to extract features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object;

[0030] The long short-term memory network is used to obtain a first feature vector of the speech emitted by the target object according to the first audio feature;

[0031] The prediction network is used to predict, based on the second audio feature, a second feature vector associated with the speech that may be uttered by the target object;

[0032] The joint network is used to identify the target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

[0033] According to the above technical means, since the mask network, long short-term memory network and prediction network in the speech recognition model are all used to extract audio features, the probability of the text is directly mapped after the features are fused based on the joint network, and the target text is obtained. After the mixed audio stream is input into the speech recognition model, the corresponding target text is directly output without multiple cascading, thereby realizing end-to-end streaming speech recognition and improving recognition accuracy.

[0034] Furthermore, after the recognition obtains the target text corresponding to the speech emitted by the target object in the mixed audio stream, the method further includes:

[0035] When it is monitored that the target text corresponding to the speech uttered by the target object at the current moment is abnormal, the model parameters of the speech recognition model are optimized based on the abnormal scenario of the target text, the application field of speech recognition and the corrected text, wherein the corrected text is the correct text corresponding to the abnormal target text.

[0036] According to the above technical means, in the process of speech recognition, abnormal speech recognition scenarios are monitored, the causes and types of recognition anomalies in the speech recognition process are learned, and they are associated with the language recognition application field. By continuously optimizing the model parameters of the speech recognition model in abnormal scenarios, the model gradually adapts to the scenarios where abnormalities may occur, reducing the number of abnormalities recognized by the speech recognition model, and thus making the model recognition more and more accurate. When enough abnormal scenarios in the speech recognition application field are learned, non-stop streaming speech recognition can be achieved. That is, based on the above technical means, the speech recognition model adapts to the dynamically changing variable-length data stream and achieves non-stop streaming recognition.

[0037] Further, the optimization of the model parameters of the speech recognition model based on the abnormal scenario of the target text, the application field of speech recognition and the correction text includes:

[0038] Training the speech recognition model according to the abnormal scenario of the target text, the application field of speech recognition and the correction text, and adjusting the loss of the speech recognition model;

[0039] adjusting forward probability algorithm model parameters and backward probability algorithm model parameters based on the adjusted loss;

[0040] Predicting the forward probability of the speech that the target object may make at a moment before the current moment based on the adjusted forward probability algorithm model parameters, and predicting the backward probability of the speech that the target object may make at a moment after the current moment based on the adjusted backward probability algorithm model parameters;

[0041] Calculating a gradient according to the forward probability, the backward probability, and the revised text;

[0042] Model parameters of the speech recognition model are adjusted based on the gradient.

[0043] According to the above technical means, the forward probability algorithm model parameters and the backward probability algorithm model parameters can be adjusted, and then the gradients are calculated to update the model parameters. On the one hand, the model parameters can be optimized, and on the other hand, the model parameters with the smallest loss can be quickly found.

[0044] Furthermore, the model parameters of the speech recognition model are optimized through the cloud.

[0045] According to the above technical means, the model parameters of the speech recognition model are optimized in the cloud, and real-time OTA can be performed on the cloud and the terminal. Deployment in the cloud can reduce the computing power of the terminal and avoid affecting the normal use of other functions of the terminal when optimizing the model parameters.

[0046] A method for training a speech recognition model, comprising:

[0047] Acquire a training sample, where the training sample includes a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream;

[0048] The sample mixed audio stream is input into a speech recognition model, and the following processing is performed in the speech recognition model to identify and obtain a target text corresponding to the speech emitted by the target object in the sample mixed audio stream: feature extraction is performed on the sample mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects other than the target object; a first feature vector of the speech emitted by the target object is obtained based on the first audio feature, and a second feature vector associated with the speech that may be emitted by the target object is predicted based on the second audio feature; the target text is identified and obtained based on the first feature vector and the second feature vector;

[0049] Calculate the loss value of the target text relative to the corresponding target sample label based on the total loss function;

[0050] The model parameters of the speech recognition model are adjusted according to the loss value.

[0051] According to the above technical means, by setting the speech recognition model to extract the features of the mixed audio stream as the first audio feature and the second audio feature during feature extraction, the mixed audio stream is distinguished, and then divided into two processing paths, the first audio feature and the second audio feature are processed respectively, wherein the first audio feature is processed to obtain the first feature vector as the first path of feature processing; the second audio feature is processed to obtain the second feature vector as the second path of feature processing, and the target text is finally recognized by fusing the first feature vector and the second feature vector. First, the mapping probability of the first audio feature and the target text is calculated through the first path, and the mapping probability of the first audio feature and the target text is predicted through the second path. The prediction results of the two paths are fused, which can improve the accuracy of speech recognition. Or it can be understood that in the same voice interaction scenario, the voices of other objects in the mixed voice stream have a certain correlation with the voice of the target object. This embodiment predicts the mapping probability of the target object's voice and the target text by adding a second feature vector associated with the voice that may be uttered by the target object, thereby improving the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scenario of the mixed audio stream, the voice recognition accuracy is high and the pauses in voice recognition during the recognition process can be reduced.

[0052] Further, the speech recognition model includes a mask network, and the feature extraction of the sample mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects other than the target object includes:

[0053] Inputting the sample mixed audio stream into the preprocessing encoder of the mask network, preprocessing the sample mixed audio stream based on the preprocessing encoder, and obtaining sample mixed audio features;

[0054] Inputting the sample audio features into the mask encoder of the mask network, processing the sample audio features based on the mask encoder, and predicting a mask of the speech emitted by the target object;

[0055] Obtaining a first audio feature of the target object according to the mask and the sample audio feature;

[0056] The complement code corresponding to the mask and the sample audio feature are used to obtain second audio features of objects other than the target object.

[0057] According to the above technical means, by performing mask processing on the mixed audio stream, the features corresponding to the voices of different objects in the mixed audio stream are distinguished, and the mixed audio stream is separated into at least two paths corresponding to the required audio features.

[0058] Further, the speech recognition model includes a long short-term memory network, and obtaining a first feature vector of the speech emitted by the target object according to the first audio feature includes:

[0059] Inputting the first audio feature into the long short-term memory network, performing feature extraction on the first audio feature based on the long short-term memory network, and obtaining a first feature vector of the speech emitted by the target object;

[0060] And / or, the speech recognition model includes a prediction network, and the predicting, based on the second audio feature, of a second feature vector of the speech that may be uttered by the target object includes:

[0061] The second audio feature is input into the prediction network, and based on the prediction network, the speech that may be uttered by the target object corresponding to the second audio feature is predicted, and a corresponding second feature vector is generated.

[0062] According to the above technical means, the training of the speech recognition model can obtain the output of the corresponding target text after the mixed audio stream is input into the speech recognition model, and the audio features are directly extracted based on the mixed audio stream, without going through multiple cascades, to achieve end-to-end speech recognition and improve recognition accuracy.

[0063] Further, the speech recognition model includes a joint network, and the recognition of the target text according to the first feature vector and the second feature vector includes:

[0064] Based on the joint network, the first feature vector and the second feature vector are fused to obtain a fused feature vector;

[0065] According to the fused feature vector, a target text corresponding to the speech uttered by the target object in the sample mixed audio stream is identified.

[0066] According to the above technical means, the correlation between the fusion of the first feature vector and the second feature vector and the text is learned to improve the accuracy of speech recognition.

[0067] Furthermore, the training method further includes:

[0068] Predicting a backward probability of a speech that the target object may utter at a next time step based on the first audio feature;

[0069] The step of identifying the target text according to the first feature vector and the second feature vector includes:

[0070] Based on the first feature vector, the second feature vector, the backward probability and the forward probability of the speech uttered by the target object predicted in the previous time step, the target text is obtained through preset conditional probability prediction.

[0071] According to the above technical means, based on the forward probability of the training sample in the previous time step corresponding to the same sequence label, and the predicted backward probability of the training sample in the next time step corresponding to the same sequence label, the correlation between the predicted probability and the forward probability and the backward probability of the training sample and the target sample label in the current time step is learned, so that the calculated loss of the target text and the target sample label is updated, thereby improving the accuracy of speech recognition.

[0072] Furthermore, the speech recognition model further includes an audio encoder and a label encoder, and the training method further includes:

[0073] Input the first audio feature into the audio encoder, encode the first audio feature based on the audio encoder, and encode the sample label according to the label encoder;

[0074] Predicting a backward probability of a speech that the target object may utter at a next time step based on the first audio feature;

[0075] Predicting the gradient of the loss function with respect to the model parameters according to the backward probability and the forward probability of the speech emitted by the target object predicted at the previous time step;

[0076] The adjusting the model parameters of the speech recognition model according to the loss value includes:

[0077] When the loss value is greater than or equal to a preset threshold, the model parameters are adjusted according to the gradient.

[0078] According to the above technical means, the gradient reflects the sensitivity of the difference (loss value) between the target text recognized by the speech recognition model and the sample label to the change of model parameters (partial derivatives of model parameters). By calculating the gradient, the model parameters are adjusted based on the gradient, so that the speech recognition model can quickly learn the relationship between the mixed audio stream and the sample label, reduce the learning samples, and improve the learning efficiency. In addition, the gradient is calculated based on the forward probability and backward probability. The guidance of the model parameters by the gradient makes the difference between the target text recognized by the speech recognition model and the sample label tend to be smaller.

[0079] Furthermore, the training method further comprises:

[0080] The total loss function is constructed based on the error between the target text and the target sample label predicted by the first audio feature and the error between the target text and the target sample label predicted by the second audio feature.

[0081] A speech recognition device, comprising:

[0082] An extraction module, used to extract features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object;

[0083] A first prediction module, configured to obtain a first feature vector of the speech emitted by the target object according to the first audio feature;

[0084] A second prediction module, configured to predict, based on the second audio feature, a second feature vector associated with a voice that may be uttered by the target object;

[0085] The recognition processing module is used to recognize and obtain the target text corresponding to the speech uttered by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

[0086] A training device for a speech recognition model, comprising:

[0087] An acquisition module is used to acquire a training sample, wherein the training sample includes a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream;

[0088] A speech recognition model is used to perform the following processing on the input sample mixed audio stream to identify the target text corresponding to the speech emitted by the target object in the sample mixed audio stream; perform feature extraction on the sample mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects other than the target object; obtain the first feature vector of the speech emitted by the target object based on the first audio feature, and predict the second feature vector associated with the speech that may be emitted by the target object based on the second audio feature; and identify the target text based on the first feature vector and the second feature vector;

[0089] A training module is used to calculate the loss value of the target text relative to the corresponding sample label based on the total loss function, and adjust the model parameters of the speech recognition model according to the loss value.

[0090] An electronic device, comprising: a memory and a processor;

[0091] The memory is used to store computer programs / instructions; the processor is used to implement the method described above according to the computer programs / instructions stored in the memory.

[0092] A vehicle, comprising: a memory, a processor;

[0093] The memory is used to store computer programs / instructions; the processor is used to implement the above-mentioned method according to the computer programs / instructions stored in the memory.

[0094] A computer-readable storage medium / computer program product, wherein the computer-readable storage medium stores a computer program / instructions, or the computer program product includes a computer program / instructions, and when the computer program / instructions are executed by a processor, they are used to implement the method as described above.

[0095] Beneficial effects of the present invention:

[0096] (1) The present invention extracts audio features by setting up dual paths. One path extracts a first sound feature quantity through the first audio feature of the target object, and the other path predicts a second feature vector corresponding to the speech that may be uttered by the target object through the second audio feature of other objects. The speech uttered by the target object is identified by using the two feature vectors. The prediction and fusion of the two feature vectors can improve the accuracy of identifying the speech uttered by the target object in the mixed audio stream.

[0097] (2) The present invention uses a speech recognition model to map an acoustic feature sequence to a label sequence and convert them synchronously, without the need for a multi-cascade model, thereby simplifying the model.

[0098] (3) The present invention calculates the partial derivatives (gradients) of the model loss function with respect to the model parameters through forward probability and backward probability, which is used to guide how to adjust the model parameters to minimize the loss function so that the speech recognition model can recognize speech with small error. Combined with the heuristic error allocation method, it can process audio streams of varying lengths, which can reduce or even avoid the occurrence of freezes.

[0099] (4) Low consumption of system resources (CPU (processor), RAM (memory), EMMC (Flash storage)) and high accuracy make the speech recognition model easy to deploy on the end side. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 An example diagram of a hardware system for a speech recognition method provided in an embodiment of the present application;

[0101] Figure 2 A schematic diagram of an application scenario of the speech recognition method provided in an embodiment of the present application;

[0102] Figure 3 A software system diagram of the speech recognition method provided in the embodiment of the present application;

[0103] Figure 4A flow chart of an embodiment of a speech recognition method provided in an embodiment of the present application;

[0104] Figure 5 A schematic diagram of the processing flow of the speech recognition model provided in the embodiment of the present application;

[0105] Figure 6 A schematic diagram of an application flow of a speech recognition method provided in an embodiment of the present application;

[0106] Figure 7 A flowchart of another embodiment of the speech recognition method provided in the embodiment of the present application;

[0107] Figure 8 A flow chart of an embodiment of a method for training a speech recognition model provided in an embodiment of the present application;

[0108] Fig. 9 A schematic diagram of the result of the speech recognition method provided in an embodiment of the present application recognizing a mixed audio stream. DETAILED DESCRIPTION

[0109] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0110] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0111] Speech recognition refers to the human-computer interaction process of converting human language into text or commands. Currently, speech recognition technology is widely used in scenarios such as voice interaction, speech transcription, speech analysis, audio and video recording, etc.

[0112] In related technologies, speech recognition is achieved by using deep learning models to train large amounts of data, learn the relationship between audio signals and corresponding texts, and thus be able to convert user speech into text.

[0113] However, in human-computer interaction scenarios, the collected voices are generally mixed voices (voices of multiple people speaking or environmental noise). Mixed voices affect recognition accuracy, and the recognition accuracy of the above-mentioned voice recognition method is low.

[0114] Based on this, an embodiment of the present application provides a speech recognition method, which recognizes speech through dual-path feature prediction results, wherein one path is based on the audio features of the target object for recognition, and the other path is based on other audio features of other objects in the mix except the target object for prediction, which can improve the recognition accuracy.

[0115] The following is a description of the hardware system, software system and specific implementation examples involved in the application of the embodiments of the present application.

[0116] Figure 1 An example diagram of the hardware system of the speech recognition method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the hardware system is applied on the device side, which includes a microphone MIC, a signal processor, a low-pass filter, a digital-to-analog converter ADC / A2B bus (ADC / A2B), an audio digital signal processor Audio DSP, a main controller AVN / HU / IVI SOC, and a communication module.

[0117] The microphone MIC is used to collect audio signals (for streaming voice, it corresponds to collecting audio signal streams). The audio signals collected by the microphone MIC are transmitted to the signal processor, which performs gain amplification (such as an active filter amplifier or a programmable amplifier) ​​and noise reduction on the audio signals. After the signal processor performs gain amplification and noise reduction on the audio signals, it transmits the audio signals to the low-pass filter, which performs low-pass filtering on the audio signals to remove high-frequency signals. The low-pass filter performs analog-to-digital conversion on the audio signals after high frequencies are removed through the digital-to-analog conversion ADC / A2B bus and converts them into audio digital signals. The audio digital signals are input into the Audio DSP and processed by the Audio DSP.

[0118] Optionally, in this embodiment, the low-pass filter filters high frequencies exceeding audio 20KHz.

[0119] Optionally, the audio signal is processed based on the above hardware system.

[0120] As an example, in the application scenario of the speech recognition method in a smart network car, the hardware system is arranged on the car. Figure 2As shown, the microphone MIC is arranged in the smart cockpit to collect the audio signals of the people speaking in the smart cockpit. When applied to the car, the driver or the passenger can realize the functions of voice recognition interaction, voice transcription, voice analysis, audio and video recording, etc. inside the car. Through voice interaction control, the driver can realize voice control of the vehicle while focusing on driving, thereby reducing the risk of accidents caused by distracted operation of the vehicle system and improving driving comfort. Therefore, the embodiment of the present application is applied to the car with better use effect.

[0121] Figure 3 An example diagram of a software system for the speech recognition method provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, the software system can be configured in the main controller of the device. The software system includes: hardware abstraction layer (HAL layer), service layer, framework layer and application layer, among which:

[0122] The Audio DSP performs digital noise reduction and data processing on the audio digital signal again, and sends it to the AVN / HU / IVI SOC (intelligent cockpit controller) system HAL (hardware abstraction layer) layer through the IIS (Inter-IC Sound) ( / PCM (Pulse Code Modulation) / TDM (Time Division Multiplexing) / PDM (Pulse Density Modulation)) bus.

[0123] In the HAL layer, the audio digital signal is processed to identify the text corresponding to the audio digital signal. Specifically, the speech feature sequence corresponding to the audio digital signal is extracted based on the dual-path modeling algorithm, and the text sequence corresponding to the speech feature sequence is identified based on the pre-learned association probability between the speech feature sequence and the text. Among them, the dual-path modeling algorithm can improve the accuracy of speech recognition, especially when there is mixed audio signal, it can distinguish the speech, and predict and recognize the distinguished speech based on other speech, thereby improving the recognition accuracy.

[0124] Furthermore, in the HAL layer, the recognition of streaming speech is also realized based on the time-synchronized sequence conversion model.

[0125] For the specific processing flow and implementation of the voice signal in the HAL layer, please refer to the following embodiments.

[0126] Figure 4 A flow chart of a speech recognition method proposed in this embodiment is shown in FIG. Figure 4 As shown, the method includes:

[0127] S401, extracting features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object;

[0128] The mixed audio stream is a stream mixed audio collected by a microphone or a microphone array, and refers to the audio collected continuously. The speech recognition method in the embodiment of the present application can process the mixed audio stream to realize real-time processing and feedback of the mixed audio.

[0129] After obtaining the mixed audio stream, feature extraction is performed on the mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects other than the target object. The target object refers to a certain speaking object in the mixed audio stream, and the mixed audio stream includes the voices of multiple objects, wherein the target object is an object distinguished from the multiple objects. It should be noted that all audio features are sorted according to the time sequence of the mixed audio stream input to form a time series, and the first audio feature refers to the audio feature corresponding to the voice emitted by the target object at a certain time step in the time series extracted from the mixed audio stream, and the second audio feature refers to the audio feature corresponding to the voice emitted by other objects at the time step, and the first audio feature and the second audio feature are audio features corresponding to the voice emitted at the same time step.

[0130] As an example, features may be extracted from a mixed audio stream based on Mel-frequency cepstral coefficients (MFCCs) to distinguish different sound sources, the mixed audio stream may be distinguished in sequence in time sequence, and the distinguished objects may be used as target objects in sequence.

[0131] For example, assume that the mixed audio stream contains the voices of two people, namely object A1 and object A2. If the audio features emitted by object A1 are recognized, object A1 is determined to be the target object, and the audio features corresponding to the voice emitted by object A1 are extracted as the first audio features, and the audio features of other objects except object A1 are extracted as the second audio features. When the voice emitted by object A2 is recognized, object A2 is taken as the target object, and the audio features corresponding to the voice emitted by object A2 are extracted as the first audio features, and the audio features of other objects except object A2 are extracted as the second audio features. The voices of each object are distinguished in turn, and the voice of each object is distinguished from the mixed audio stream, and then the voice of each object is recognized.

[0132] Assume that the mixed audio stream includes the voices of three or more people, namely object A1, object A2, ... object An. If the audio features emitted by object A1 are recognized, object A1 is determined to be the target object, the audio features corresponding to the voice emitted by object A1 are extracted as the first audio features, and the audio features of other objects (A2~An) except object A1 are extracted as the second audio features. At this time, based on the second audio features or the mixed audio features, the mixed audio stream is updated with the second audio features, and the voice in the mixed audio stream is continued to be recognized.

[0133] As an example, see Figure 5 The speech recognition method in the embodiment of the present application adopts an end-to-end speech recognition model for recognition. In this example, the speech recognition method is applied to a computing device deployed with a speech recognition model, wherein the speech recognition model includes a mask network, wherein:

[0134] The mask network is used to extract features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object.

[0135] This embodiment extracts audio features of different objects through a mask network to distinguish audio features of different objects. Optionally, the mask network includes a mask encoder, which can predict a mask based on a mixed audio stream, and the mask can reflect the contribution of different speakers in the mixed audio stream, thereby highlighting the audio features of one object through the mask and suppressing the audio features of another object. Therefore, the audio features of one object can be extracted through the mask to distinguish audio features of different objects.

[0136] Optionally, S401 may specifically be:

[0137] Predicting a mask of speech produced by a target subject based on a mixed audio stream;

[0138] The mixed audio stream is processed according to the mask, the mixed audio stream, and the complement corresponding to the mask to obtain a first audio feature of the target object and a second audio feature of other objects other than the target object.

[0139] When training the speech recognition model, the mask encoder learns predictive coding to distinguish the various sound sources in the mixed audio stream. Therefore, the mask encoder can predict the mask of the speech emitted by the target object, and extract the features of the mixed audio stream through the mask to obtain the first audio feature. The audio features other than the first audio feature are the second audio features.

[0140] For example, a mask M corresponding to the speech emitted by the target object is predicted based on the mixed audio stream. The mask value can be [0,1], and the complement corresponding to the mask is 1-M. The first audio feature corresponding to the target object is determined by the mask. Specifically, the first audio feature is H1, then ; Correspondingly, the second audio feature corresponding to other objects other than the target object is determined by the mask, and the specific second audio feature is H2, then ,in, It is a mixed audio stream, or mixed audio features after preprocessing of the mixed audio stream, for example:

[0141] As an example, in order to further improve the accuracy of feature extraction, after obtaining the mixed audio stream, the mixed audio stream is preprocessed to obtain mixed audio features. The preprocessing includes converting the mixed audio stream into a format or feature set for further processing.

[0142] Alternatively, in the example of the speech recognition model, the mask network also includes a preprocessing encoder (MixEnc), which processes the mixed audio stream using the preprocessing encoder to obtain mixed audio features.

[0143] Optionally, the processing of the preprocessing encoder includes: framing, window function, fast Fourier transform, power spectral density, Mel filter bank, logarithmic compression, pooling, deep learning feature extraction, batch normalization, activation function, etc., to extract features and obtain mixed audio features.

[0144] As an example, the preprocessing encoder preprocesses the mixed audio stream by the following formula:

[0145] ;

[0146] Where, t is time; For continuous audio; is the window function; For Mel filter at frequency Corresponding to the Mel frequency response; ε is a constant; σ 2 is the variance; μ is the batch mean.

[0147] It should be noted that in the above formula, is the activation function; is the pooling function; log is the logarithm; STFT is the short-time Fourier transform (Short-Time Fourier Transform).

[0148] After preprocessing the mixed audio stream based on the above formula, we get MixkEnc(X), and the corresponding mixed audio feature for:

[0149] ;

[0150] As an example, the mask encoder encodes the mixed audio features by the following formula Perform masking:

[0151] ;

[0152] in, for Audio features output by the convolutional layer; The input features (audio features) received by the mask encoder; is the activation function; It is a convolutional neural network; is the weight of the fully connected layer is the bias of the fully connected layer.

[0153] It should be noted that in the above formula, the MaskEnc convolutional neural network (CNN) is used to learn the feature representations of different speakers in the mixed audio stream.

[0154] After preprocessing the mixed audio stream based on the above formula, MaxkEnc(X) is obtained, and the corresponding prediction expression of the target object mask is:

[0155] ;

[0156] Where X is the mixed audio feature, It is a sigmoid activation function, which is used to map the output of MaskEnc to the interval (0, 1) to ensure that the element values ​​of the mask are reasonable.

[0157] The mask is a two-dimensional weight matrix. The first audio feature is obtained by element-by-element multiplication of the mask and the mixed audio stream. The second audio feature is obtained by element-by-element multiplication of the complement and the mixed audio stream. The first audio feature represents the audio feature of the target object, and the second audio feature represents the audio feature of other objects other than the target object.

[0158] In this embodiment, when the mixed audio stream is preprocessed, the mixed audio stream is also time-series encoded. The first audio feature obtained by the mask and the mixed audio stream and the second audio feature obtained by the complement and the mixed audio stream are time-synchronized features. This embodiment performs identification based on the time-synchronized first audio feature and the second audio feature.

[0159] S402, obtaining a first feature vector of the speech emitted by the target object according to the first audio feature;

[0160] In this step, by processing the first audio feature, the correlation relationship between the first audio feature and the context, the dependency relationship with the text, and the time series, etc. are obtained to form a first feature vector, so that the corresponding text can be more accurately identified through the first feature vector.

[0161] As an example, see Figure 5 The speech recognition model also includes a long short-term memory network, which is used to obtain a first feature vector of the speech emitted by the target object according to the first audio feature. The first audio feature is recognized, classified, and processed in real time through the long short-term memory network, and the time dependency and change law are captured. The first audio feature is modeled and predicted, and continuously and dynamically processed and recognized to realize the recognition of streaming speech.

[0162] S403, predicting a second feature vector associated with a voice that may be uttered by the target object based on the second audio feature;

[0163] In this step, the second audio feature is predicted to predict a second feature vector associated with the speech that may be uttered by the target object. The second feature vector also includes the association relationship between the second audio feature and the context, the dependency relationship with the text, and the time series.

[0164] The second audio feature is associated with the speech (first audio feature) that may be emitted by the target object: the first audio feature and the second audio feature are audio features corresponding to the same time step. In the same scene, the speech emitted by different objects has a certain association relationship. This embodiment defines the association relationship as having an auxiliary effect on the recognition of the speech that may be emitted by the target object, and is an associated feature with a positive effect. For example, in a conference scene, different objects speak on the same topic, and the speech emitted in the same time step has an association relationship. This embodiment is applied to a mixed audio stream, including the speech emitted by the A1 object and the A2 object. Based on the fact that the speech content of the A1 object and the A2 object must have an association relationship, the association relationship between the speech of different objects in the mixed audio stream in the same scene can be set. After the audio feature of the A1 object is identified as the first audio feature, the audio feature of the A2 object is used to infer the speech that may be emitted by the A1 object, and then the second feature vector corresponding to the speech is obtained.

[0165] For example, if object A1 is the target object, the text corresponding to the first audio feature may be "you", and if the text corresponding to the second audio feature may be 'hi', then the speech that object A1 may make is predicted to be "you", and then the first feature vector and the second feature vector are combined to increase the recognition accuracy of the first audio feature. Similarly, if A2 is the target object, feature processing is also performed in the above manner.

[0166] Alternatively, the mixed audio stream includes speech emitted by objects A1 and A2. The corresponding text probability is predicted by identifying the audio features of the A1 object, the corresponding text probability is predicted by identifying the audio features of the A2 object, and then the text probability is predicted by the audio features of the A2 object to predict the probability of text that may be emitted by A1. The text probability predicted by identifying the audio features of the A1 object and the text probability predicted by the A2 object are combined to predict the probability of the final target text. The prediction method in this example is similar to or equivalent to the method exemplified above.

[0167] For example, if object A1 is the target object, the text corresponding to the first audio feature may be "you", and if the text corresponding to the second audio feature may be 'hi', then the first feature vector corresponding to the voice of "you" and the second feature vector corresponding to the voice of "hi" are combined to increase the recognition accuracy of the first audio feature. Similarly, if A2 is the target object, feature processing is also performed in the above manner.

[0168] Optionally, the association relationship between the second audio feature and the speech that may be uttered by the target object (the first audio feature) may also be: the second audio feature may also be the background noise corresponding to the speech that may be uttered by the target object. By identifying the background noise, the difference between the speech that may be uttered by the target object is increased, thereby improving the accuracy of speech recognition.

[0169] As an example, see Figure 5 The speech recognition model also includes a prediction network, which is used to predict a second feature vector associated with the speech that may be uttered by the target object based on the second audio feature. In the process of processing the second audio feature by the prediction network, based on time series synchronization processing, the speech that may be uttered by the target object at the same time step is predicted, thereby improving the accuracy of speech recognition.

[0170] S404: Identify and obtain a target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

[0171] As an example, the first text prediction probability corresponding to the first feature vector can be obtained through the mapping relationship between the first feature vector and the text; the second text prediction probability corresponding to the first feature vector can be obtained through the mapping relationship between the second feature vector and the text, and then the first text prediction probability and the second text prediction probability are combined to determine the target text.

[0172] As an example, the first feature vector and the second feature vector may be fused to obtain a fused feature vector; and then the target text may be recognized based on the fused feature vector.

[0173] As an example, the speech recognition model also includes a joint network, which is used to identify the target text corresponding to the speech uttered by the target object in the mixed audio features based on the first feature vector and the second feature vector.

[0174] As an example, an embodiment of the present application processes a mixed audio stream to recognize the speech of different objects respectively. For example, when the mixed audio stream includes the speech of two people, an object is determined as the target object. When the target text corresponding to the speech emitted by the target object is recognized through the above S401~S404, the second audio feature is used as the mixed audio feature, and the above S401~S404 process is repeated to determine another object as the target object and recognize the target text corresponding to the other object. When the mixed audio stream includes the speech of three or more people, an object is determined as the target object. When the target text corresponding to the speech emitted by the target object is recognized through the above S401~S404. Then, based on the second audio feature, the mixed audio feature is updated, and the above S401~S404 process is repeated to determine another object as the target object and recognize the target text corresponding to the other object. Then, the mixed audio feature is continued to be updated based on the second audio feature, and the above S401~S404 process is repeated to determine another object as the target object and recognize the target text corresponding to the other object, until the speech of all objects in the mixed audio stream is recognized separately, and texts corresponding to different objects are generated. Fig. 9 As shown, X1 is the voice emitted by object 1 in the mixed audio stream, and Y1 is the recognized text corresponding to the voice emitted by object 1; X2 is the voice emitted by object 2 in the mixed audio stream, and Y2 is the recognized text corresponding to the voice emitted by object 2, and the texts corresponding to each object are output respectively.

[0175] In an embodiment of the present application, when extracting features from a mixed audio stream, the features of the mixed audio stream are extracted into a first audio feature and a second audio feature to distinguish the mixed audio stream, and then divided into two processing paths, and the first audio feature and the second audio feature are processed respectively, wherein the first audio feature is processed to obtain a first feature vector as the first path for feature processing; the second audio feature is processed to obtain a second feature vector as the second path for feature processing, and the target text is finally identified by fusing the first feature vector and the second feature vector. First, the mapping probability of the first audio feature and the target text is calculated through the first path, and the mapping probability of the first audio feature and the target text is predicted through the second path. The prediction results of the two paths are fused to improve the accuracy of speech recognition. Or it can be understood that in the same voice interaction scenario, the voices of other objects in the mixed voice stream have a certain correlation with the voice of the target object. This embodiment predicts the mapping probability of the target object's voice and the target text by adding a second feature vector associated with the voice that may be uttered by the target object, thereby improving the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scenario of the mixed audio stream, the voice recognition accuracy is high and the pauses in voice recognition during the recognition process can be reduced.

[0176] Figure 6 A schematic diagram of an application flow of a speech recognition method provided in an embodiment of the present application is provided as an example. Figure 6 As shown, the speech recognition scenarios in this example include but are not limited to speech control recognition, speech transcription, speech analysis, and audio and video recording. Different speech recognition scenarios correspond to different ways of outputting the target text. Based on this, in the actual application process, based on the selected speech recognition scenario, the first feature vector and the second feature vector are processed to obtain the target text corresponding to the speech emitted by the target object in the mixed audio stream, and the target text is responded to according to the recognition scenario. For example, if it is a speech control recognition scenario, the first feature vector and the second feature vector are identified based on the speech recognition scenario, and the target text reflecting the control instruction is obtained, and the control instruction corresponding to the target text is executed.

[0177] In the example, the mixed audio stream is recognized in combination with the specific scenario of speech recognition. During the recognition process, based on the attributes or features of the speech recognition scenario, the probability of text matching the corresponding attributes or features can be increased, which can further improve the accuracy of speech recognition.

[0178] Figure 7 Schematic diagram of the process of the speech recognition method provided in the embodiment of the present application Figure 2, this embodiment is based on the above embodiment. Since streaming voice technology can process and feedback in real time, it improves the efficiency of human-computer interaction and user experience, and is a development trend of voice recognition. But it also faces challenges in technical implementation and application scenarios. For example, in the scenario of model-based voice recognition, the longer the input voice, the greater the error of model recognition will be. When the recognition error is large, the model cannot continue to recognize the voice. Therefore, in the related technology, after recognizing audio of a certain length, it is necessary to pause for a period of time and then re-recognize. For example, after recognizing a 30s audio stream, there will be a pause, and the pauses are frequent. Based on this, the embodiment of the present application reduces the number of pauses and extends the recognition length of the voice stream through the following embodiments. As Figure 7 As shown, the identification method includes:

[0179] S701, extracting features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object;

[0180] S702, obtaining a first feature vector of the speech emitted by the target object according to the first audio feature;

[0181] S703, predicting a second feature vector associated with a voice that may be uttered by the target object based on the second audio feature;

[0182] In this embodiment, the specific implementation process of steps S501 to S503 is the same as above. Figure 4 The implementation process of S401 to S403 in the embodiment is similar and will not be repeated here.

[0183] S704, predicting the backward probability of the speech that the target object may utter in the next time step according to the first audio feature;

[0184] In this step, the forward probability refers to the probability that the speech that the target object may emit in the previous time step (t-1) corresponds to the first text, predicted at the current time step t. The backward probability refers to the probability that the speech that the target object may emit in the next time step (t+1) corresponds to the second text, predicted based on the first audio feature in the current time step t, wherein the second text may include multiple, and the backward probability of each second text is different. The first text, the target text, and the second text are the texts corresponding to the speech emitted by the target object successively, and constitute the continuous text corresponding to the speech stream of the target object.

[0185] There is no restriction on the execution order between this step and S702 and S703, and they can be executed one after the other or simultaneously.

[0186] S705 , obtaining a target text corresponding to the speech emitted by the target object in the mixed audio stream through a preset conditional probability prediction according to the first eigenvector, the second eigenvector, the backward probability and the forward probability.

[0187] As an example, according to the first eigenvector, the second eigenvector, the backward probability and the forward probability, the target text corresponding to the speech emitted by the target object in the mixed audio stream is obtained by predicting the preset conditional probability, specifically:

[0188] Fusing the first eigenvector and the second eigenvector to obtain a fused feature vector;

[0189] Predict the backward probability of the speech that the target object may make at the next time step based on the fused feature vector;

[0190] Based on the forward probability predicted by the target object in the previous time step and the backward probability of the speech that may be uttered in the next time step, the target text corresponding to the speech uttered by the target object in the mixed audio stream is predicted through the preset conditional probability.

[0191] As an example, the preset conditional probability is:

[0192] ;

[0193] in, is the forward probability, is the backward probability, P(X) is the normalization constant; The probability of predicting the target text based on the first feature vector and the second feature vector.

[0194] The target text corresponding to the speech is obtained through probability. For example, in the above conditional probability, X is the speech and Y is the target text. is the probability of the speech corresponding to the text. If the probability of the speech corresponding to a text is the highest, the text is determined to be the target text. For example, if the probability of the speech corresponding to "hello" is 98%, then the target text corresponding to the speech is "hello".

[0195] During the model training process, an association relationship between conditional probability and forward probability and backward probability is established (such as the above-mentioned conditional probability). In actual application, after the fused feature vector is obtained based on the fusion of the first feature vector and the second feature vector, the fused feature vector is used to predict the backward probability of the speech that the target object may utter at the next time step. Then, the probability of the target text is predicted through the above-mentioned conditional probability, that is, the probability of the speech emitted at the current time step and the target text. The forward probability predicted at the previous time step and the backward probability of the speech that may be emitted at the next time step are used to predict. Based on the strong correlation between the previous and next speech of streaming speech, this prediction method can improve the prediction accuracy of the current time step and improve the recognition accuracy.

[0196] Let's take an example to illustrate the relationship between the forward probability, backward probability and the prediction result at the current moment. Assume that the mixed audio stream includes audio at time steps t1, t2, t3... tn. Suppose the target text predicted at time step t1 is "big"; when recognizing the audio at time step t2, the forward probability at this time is Q1. Based on the fused feature vector based on the first audio feature and the second audio feature, the backward probability Q2 of the speech that the target object may make at the next time step (t3) is predicted, and then the probability of the target text predicted based on Q1 and Q2 is P2. For example, the probability that Q1 is "big" is the highest, and the target text corresponding to the fused feature vector may be "home". Through this feature vector, it is predicted that the text corresponding to the next speech of the target object may be "good" with the highest probability. Then, based on the forward probability Q1 and the backward probability Q2, the prediction probability that the target text at the current time step t2 is "home" is increased. When recognizing the audio at time step t3, P2 predicted based on time step t2 is the forward probability Q1 at time step t3. Based on the fused feature vector of the first audio feature and the second audio feature, the backward probability Q2 that the speech that the target object may emit at the next time step corresponds to the text is predicted. The probability of the target text measured based on Q1 and Q2 is P3, which is the forward probability corresponding to time step t4. By analogy, when recognizing the audio at time step tn, Pn-1 predicted based on time step t-1 is the forward probability Q1 at time step tn. Based on the fused feature vector of the first audio feature and the second audio feature, the backward probability Q2 that the speech that the target object may emit at the next time step corresponds to the second text is predicted. The probability of the target text measured based on Q1 and Q2 is Pn.

[0197] It should be noted that at time step t1, the forward probability Q1 is zero. At time step tn, the backward probability Q2 is zero, or based on the time sequence coding of the mixed audio stream, when it is determined that there is no more audio input after time step tn, there is no need to predict the forward probability of the second text corresponding to the speech that the target object may emit at the next time step of time step tn.

[0198] In this embodiment, first, based on the fused feature vector of the first feature vector and the second feature vector, the backward probability of the text corresponding to the speech that may be emitted at the next time step of the current time step is predicted, and then the forward probability and the backward probability are used to predict the target text, and the prediction accuracy is improved. Since the backward probability of the text at the next time step is predicted in advance at the current time step, the speech recognition accuracy is high, and the speech recognition model can continue to recognize, reducing the pause frequency.

[0199] As an example, based on all the above embodiments, after identifying the target text corresponding to the speech emitted by the target object in the mixed audio stream, the method further includes:

[0200] When it is detected that the target text corresponding to the speech uttered by the target object at the current moment is abnormal, the model parameters of the speech recognition model are optimized based on the abnormal scenario of the target text, the application field of speech recognition and the correction text.

[0201] The corrected text is the correct text after correction when the target text is abnormal.

[0202] Target text anomalies include but are not limited to the following scenarios:

[0203] 1. In the voice control scenario, after the target text is recognized, the control instruction indicated by the target text is sent to the corresponding execution end, but the corresponding execution end does not feedback the response information; or the execution end that feedbacks the response information does not match the execution end indicated by the voice (user feedback is required in this case), which means the target text is abnormal.

[0204] Taking the voice control scenario applied in cars as an example, assuming that the mixed audio stream includes the voice commands for turning on the air conditioner and closing the car windows, the speech recognition model recognizes that the target text corresponding to the mixed audio stream is turning on the air conditioner and locking the car doors, and the car performs the actions of turning on the air conditioner and locking the car doors. At this time, the user will manually or through voice feedback related information based on the fact that the control performed by the car is different from the voice control command issued by the user, and the target text is detected to be abnormal.

[0205] 2. In the speech-to-text scenario, if the target text is in an unconventional font or is significantly different from the previous and following texts, the target text is considered abnormal.

[0206] 3. In the speech-to-text scenario, if the target text does not match the speech recognition application field, for example, in the vehicle application scenario, medical terms appear in the text, then the target text is abnormal.

[0207] Furthermore, the model parameters of the speech recognition model are optimized in combination with information such as the abnormality type and the abnormality cause, so that the recognition result is more accurate after the model parameters are optimized.

[0208] Furthermore, the embodiment of the present application optimizes the model parameters of the speech recognition model in the following manner:

[0209] According to the abnormal scenario of the target text, the application field of speech recognition and the corrected text, the speech recognition model is trained and the loss of the speech recognition model is adjusted. The corrected text can be fed back by the user or predicted. The corrected text is used as a label to train the speech recognition model, and the loss of the speech recognition model is adjusted. Optionally, the loss of the adjusted speech recognition model becomes smaller, which can improve the recognition accuracy of the speech recognition model.

[0210] The forward probability algorithm model parameters and the backward probability algorithm model parameters are then adjusted based on the adjusted loss.

[0211] Optionally, the relationship between the loss of the speech recognition model and the forward probability and backward probability is:

[0212] ;

[0213] When Lrnnt(X, Y) is adjusted, the forward probability Algorithm model parameters, backward probability The algorithm model parameters are adjusted.

[0214] Predicting the forward probability of the speech that the target object may make at the moment before the current moment based on the adjusted forward probability algorithm model parameters, and predicting the backward probability of the speech that the target object may make at the moment after the current moment based on the adjusted backward probability algorithm model parameters;

[0215] Optionally, the forward probability is predicted by:

[0216] .

[0217] Optionally, the backward probability is predicted by:

[0218] .

[0219] Calculate the gradient based on the forward probability, backward probability, and corrected text;

[0220] Optionally, the gradient is calculated as follows:

[0221] .

[0222] Furthermore, model parameters of the speech recognition model are adjusted based on the gradient.

[0223] This embodiment uses the above method to quickly optimize the model parameters of the speech recognition model.

[0224] Optionally, in this embodiment, if Figure 6 As shown in the figure, the model parameters of the speech recognition model are optimized through the cloud. The model parameters of the speech recognition model are optimized in the cloud. The cloud and the terminal can be OTA in real time. Deployment in the cloud can reduce the computing power of the terminal and avoid affecting the normal use of other functions of the terminal when optimizing the model parameters.

[0225] It is understandable that the speech recognition model in this application is set based on the heuristic error allocation method, and uses heuristic rules to quickly select the optimal model parameters. Specifically, when training the model, a basic model is pre-trained through limited annotated data. Compared with the conventional model training that relies on complete annotated data, this application can reduce the data acquisition cost and reduce the difficulty of training.

[0226] The basic model is then applied to the end, and the recognition results of the model are used during the use of the end to measure whether the current solution of the model is optimal.

[0227] Through statistical analysis of application fields and error patterns, according to error types, causes, scenarios, etc., based on heuristic rules, new model parameters are obtained. If the model parameters make the recognition results of the speech recognition model more accurate, that is, the current solution of the model is better, the speech recognition model is updated. In the continuous process, the model is gradually improved based on the heuristic error training method, and the same errors are learned and reduced in the correction, gradually improving the accuracy of the speech recognition model. In the same application field, through continuous learning, to a certain extent, non-stop streaming speech recognition can be achieved.

[0228] In this embodiment, the way in which the heuristic rule obtains new model parameters can refer to the method listed above, that is, the gradient is calculated through the forward probability and the backward probability, and then the modification direction of the model parameters is reflected based on the gradient, so as to obtain the modified model parameters.

[0229] In this embodiment, when an abnormality occurs in the recognition of the target text, the model parameters of the speech recognition model are optimized through the abnormal scene of the target text, the application field of speech recognition, and the correction text to improve the recognition accuracy of the speech recognition model. It should be noted that in the application process, the embodiment of the present application is based on the recognition of abnormalities, that is, it starts to learn how to obtain the minimum loss of the speech recognition model under abnormal circumstances, and then continuously corrects the speech recognition model, rather than re-correcting or re-recognizing the speech after the gradient of the speech recognition model explodes. Therefore, compared with the related art, the embodiment of the present application can make the accuracy of the speech recognition model higher and higher based on the continuous optimization and correction of the speech recognition model, and then can recognize indefinite length streaming speech.

[0230] Moreover, if the speech recognition model is migrated to a new application field, the model can be gradually adjusted to adapt to the characteristics of the new field based on the heuristic error allocation rule, making the speech recognition model easy to adapt to various fields.

[0231] The speech recognition model mentioned in the above embodiment is obtained after training by the speech recognition model training method provided in the embodiment of the present application. Figure 8 In this embodiment, the training method includes:

[0232] S801, obtaining a training sample, where the training sample includes a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream;

[0233] In this step, the sample label is the actual target text (hereinafter collectively referred to as the sample label), which can be a text annotation of the sample mixed audio stream or a text set according to the mixed sample audio stream.

[0234] S802, input the sample mixed audio stream into the speech recognition model, and perform the following processing in the speech recognition model to identify the target sample label corresponding to the speech emitted by the target object in the sample mixed audio stream: perform feature extraction on the sample mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects other than the target object; obtain the first feature vector of the speech emitted by the target object based on the first audio feature, and predict the second feature vector associated with the speech that may be emitted by the target object based on the second audio feature; and identify the target sample label based on the first feature vector and the second feature vector.

[0235] In an embodiment of the present application, the speech recognition model includes a mask network, a long short-term memory network, a prediction network and a joint network. The sample mixed audio stream is feature extracted through the mask network to extract the first audio features and the second audio features of different people, and two prediction paths are formed through the long short-term memory network and the prediction network, wherein the long short-term memory network is the first path (main path), which is used to predict the corresponding first feature vector based on the first audio feature of the target object, and the prediction network is the second path (auxiliary path), which is used to predict the second feature vector of the speech that the target object may emit based on the second audio feature. That is, the long short-term memory network and the prediction network form a dual prediction path, so that the speech recognition model is a model formed by dual-path modeling. After obtaining the first feature vector and the second feature vector based on the dual prediction path, the first feature vector and the second feature vector are fused and predicted through the joint network to obtain the probability of the target text.

[0236] Based on the structure of the speech recognition model, after the sample mixed audio stream is input into the speech recognition model, the sample mixed audio stream is first input into the mask network, and the sample mixed audio stream is masked by the mask network. Specifically, in some embodiments, the mask network includes a mask encoder, the sample mixed audio stream is input into the mask encoder, the sample mixed audio stream is processed based on the mask encoder, and the mask of the speech emitted by the target object is predicted; the sample mixed audio stream is processed according to the mask, the sample mixed audio stream and the complement corresponding to the mask, and the first audio feature of the target object and the second audio feature of other objects other than the target object are obtained to distinguish the audio features of different objects.

[0237] Alternatively, in some embodiments, the mask network includes a preprocessing encoder and a mask encoder, the sample mixed audio stream is input into the preprocessing encoder, and the sample mixed audio stream is first preprocessed based on the preprocessor encoder to obtain the sample mixed audio features. Then the sample mixed audio features are input into the mask encoder, and the sample mixed audio features are processed based on the mask encoder to predict the mask of the speech emitted by the target object; the sample mixed audio features are processed according to the mask, the sample mixed audio features and the complement corresponding to the mask to obtain the first audio features of the target object and the second audio features of other objects other than the target object, so as to distinguish the audio features of different objects.

[0238] As an example, based on the mask and the sample mixed audio feature, the first audio feature of the target object is obtained; based on the complement and the sample mixed audio feature, the second audio feature is obtained.

[0239] After the sample mixed audio stream passes through the mask network, the feature extraction of the sample mixed audio stream is completed, and the feature extraction is performed as the first audio feature and the second audio feature. Then the first audio feature is input into the long short-term memory network, and the first audio feature is extracted based on the long short-term memory network to obtain the first feature vector of the speech emitted by the target object. The second audio feature is input into the prediction network, and the speech that the target object corresponding to the second audio feature may emit is predicted based on the prediction network to generate the corresponding second feature vector. By processing the first audio feature and the second audio feature respectively through dual paths, the audio features of a single target object in the mixed audio stream can be separated to obtain the first feature vector, and the second feature vector corresponding to the speech that the target object may emit can be predicted based on the second audio features of other objects in the mixed audio stream. The second feature vector increases the probability between the first feature vector and the target text, thereby improving the recognition accuracy.

[0240] The first audio feature is passed through a long short-term memory network to obtain a first feature vector, and the second audio feature is passed through a prediction network to obtain a second feature vector. The first feature vector and the second feature vector are then respectively input into a joint network, and the first feature vector and the second feature vector are fused based on the joint network to obtain a fused feature vector. Based on the fused feature vector, the target text corresponding to the speech emitted by the target object in the sample mixed audio stream is identified.

[0241] As an example, before the first audio feature is input into the long short-term memory network, the first audio feature and the sample label need to be encoded to facilitate the time synchronization conversion of the mapping audio feature and the sample label. Therefore, the speech recognition model also includes an audio encoder and a label encoder. The audio encoder is used to encode the first audio feature, and the label encoder is used to encode the sample label (that is, the label sample). Among them, the audio encoder encodes the first audio feature based on the time sequence, and the label encoder also encodes the sample label based on the time sequence, so as to obtain the first audio feature and sample label with time sequence synchronization. For example, the audio feature corresponding to the mixed audio stream is X={x1,x2 ..., x t}, the corresponding sample label Y={y1,y2 ...,y u}, t is the temporal length of the mixed audio stream, and u is the length of the sample label.

[0242] In this embodiment, the speech recognition model processes the sample mixed audio stream, and the sample label is converted synchronously based on the time series. In the process of time synchronization, the original time series data is converted into a feature vector representation. The long short-term memory network (LSTM) recognizes, classifies, and processes the data in real time, captures the time dependency and change rules, models and predicts the time series data, and continuously and dynamically processes and recognizes it, which can improve the accuracy and reliability of the prediction. It provides the possibility for the speech recognition model to be deployed at the end.

[0243] S803, calculating the loss value of the target text relative to the corresponding sample label based on the total loss function;

[0244] S804, adjusting the model parameters of the speech recognition model according to the loss value.

[0245] It should be noted that in the embodiment of the present application, the mapping relationship between the target text and the mixed audio stream is represented by the association probability. For example, the association probability between the speech emitted by the target object in the mixed audio stream at the current time step and the text "you" is A%, which is the maximum probability, and the text "you" is determined to be the target text corresponding to the speech emitted by the target object at the current time step. The embodiment of the present application calculates the loss value of the target text relative to the corresponding sample label through the total loss function, and then continuously adjusts the size of the association probability based on the loss function value, so that the recognition accuracy of the speech recognition model is high.

[0246] As an example, a total loss function is constructed based on the error between the target text and the target sample label predicted by the first audio feature and the error between the target text and the target sample label predicted by the second audio feature.

[0247] Optionally, the total loss function is:

[0248] ;

[0249] Where X is the mixed audio stream, H1 is the first audio feature, H2 is the second audio feature, Y1 is the first sample label corresponding to the first audio feature, and Y2 is the second sample label corresponding to the second audio feature; Lrnnt is the loss function between the input audio feature H and the corresponding sample label Y; L heat is the loss value.

[0250] It should be noted that the total loss function reflects a heuristic error allocation method. In the heuristic error allocation (selecting a label allocation based on heuristic information), the model is trained to minimize L heat The model learns to match the feature representations H1 and H2 with their corresponding label sequences (target text sequences) Y1 and Y2 respectively. The mixed audio stream X obtains H1 and H2 based on mask processing, and the error between the target text predicted by H1 and the first sample label Y1, and the error between the target text predicted by H2 and the second sample label Y2 are calculated through probability conditions. The total loss function reflects the sum of the errors.

[0251] When the loss value is greater than or equal to a preset threshold, the model parameters of the speech recognition model are adjusted so that the loss value can be reduced when learning the next training sample, thereby improving the recognition accuracy of the speech recognition model.

[0252] In an embodiment of the present application, the speech recognition model is set to extract the features of the mixed audio stream as the first audio feature and the second audio feature during feature extraction, so as to distinguish the mixed audio stream, and then divide it into two processing paths, and process the first audio feature and the second audio feature respectively, wherein the first audio feature is processed to obtain the first feature vector as the first path of feature processing; the second audio feature is processed to obtain the second feature vector as the second path of feature processing, and the target text is finally recognized by fusing the first feature vector and the second feature vector. Based on learning the association between the first audio feature and the text, and simultaneously learning the association between the second audio feature and the text, the probability of learning the relationship between the mixed audio stream and the text is increased through two paths, and the recognition accuracy of each target object in the mixed audio stream can be improved.

[0253] In a further embodiment, the training method of the speech recognition model further includes: predicting, based on the first audio feature, a backward probability that the speech that the target object may emit at the next time step corresponds to the second sample label;

[0254] Based on the first feature vector, the second feature vector, the backward probability and the forward probability that the speech emitted by the target object corresponds to the first sample label predicted in the previous time step, the target text is obtained through a preset conditional probability prediction.

[0255] That is, in this embodiment, the first eigenvector, the second eigenvector, the backward probability and the forward probability are combined to comprehensively identify the target text corresponding to the speech uttered by the target object in the sample mixed audio stream to improve the accuracy of speech recognition.

[0256] Optionally, the preset conditional probability is:

[0257] ;

[0258] in, is the forward probability, is the backward probability, P(X) is the normalization constant, The probability of predicting the target text based on the first feature vector and the second feature vector.

[0259] In this example, the RNNT model of the audio encoder, label encoder, long short-term memory network (LSTM), and joint network is used to model the conditional probability relationship between audio features (acoustic features) and sample labels (target text), directly mapping the acoustic feature sequence to the label sequence (target text sequence) and converting them synchronously. Both the training and decoding processes of the model rely on the calculation of conditional probabilities.

[0260] As an example, the speech recognition model also includes an audio encoder and a label encoder, and the training method also includes:

[0261] Obtaining a training sample, where the training sample includes a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream;

[0262] In this step, the sample label is the actual target text (hereinafter collectively referred to as the sample label), which can be a text annotation of the sample mixed audio stream or a text set according to the mixed sample audio stream.

[0263] The sample mixed audio stream is input into a speech recognition model, and the following processing is performed in the speech recognition model to identify the target text corresponding to the speech emitted by the target object in the sample mixed audio stream: feature extraction is performed on the sample mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects other than the target object; based on the first audio feature, a first feature vector of the speech emitted by the target object is obtained, and based on the second audio feature, a second feature vector associated with the speech that may be emitted by the target object is predicted; based on the first feature vector and the second feature vector, the target text is identified.

[0264] Based on the above content, in the embodiment of the present application, the training process of the speech recognition model can be: the system is powered on, the MIC receives the mixed acoustic sequence in the cabin, and the preprocessing encoder (MixEnc) extracts features. The mask encoder (MaskEnc) estimates a mask M. Through the mask M and the preprocessed features, the feature representations H1 and H2 of the two speakers of the overlapping audio can be calculated, and the overlapping audio speech acoustic feature sequence X={x1,x2 ..., x t} and its corresponding label sequence (target text sequence) Y={y1,y2 ...,y u}, where t is the length of the acoustic sequence and u is the length of the label sequence (target text sequence). Convert the label sequence (target text sequence) Y into an embedded representation of the label encoder. Dual-path modeling captures both the local features and global context information of the speech signal through the sound features and temporal dependencies between human voices in single-person and multi-person audio environments and under different signal-to-noise ratio conditions. The audio encoder and label encoder network calculate the forward probability of each time step. and the backward probability at each time step . Calculate the probabilities of all possible label sequence (target text sequence) paths (including blank tokens). Calculate the normalized probability distribution through forward and backward probabilities. The RNNT model of the audio encoder, label encoder, long short-term memory network (LSTM) and joint network models the conditional probability relationship between acoustic features and label sequences (target text sequences) with time-synchronized sequence transformation, maps the acoustic feature sequence to the label sequence (target text sequence) and simultaneously converts the audio features into text to predict the corresponding label sequence (target text sequence).

[0265] As an example, the speech recognition model further performs the following processing: inputting the first audio feature into the audio encoder, encoding the first audio feature based on the audio encoder, encoding the sample label according to the label encoder, predicting the backward probability of the second sample label corresponding to the speech that the target object may emit at the next time step according to the first audio feature; predicting the gradient of the model parameter corresponding to the loss function according to the backward probability and the forward probability that the speech emitted by the target object predicted at the previous time step corresponds to the first sample label;

[0266] As an example, the gradient is calculated by the following formula:

[0267] ;

[0268] in, and are model parameters respectively; Lrnnt is the loss function between the input audio feature H and the corresponding sample label Y; is the forward probability, is the backward probability.

[0269] The loss value of the target text relative to the corresponding sample label is calculated based on the total loss function; this step is not restricted in order with the above gradient prediction step, and can be performed simultaneously or one after another.

[0270] When the loss value is greater than or equal to a preset threshold, the model parameters of the speech recognition model are adjusted according to the gradient.

[0271] In this example, the gradient reflects the sensitivity of the difference (loss value) between the target text and the sample label recognized by the speech recognition model to the change of model parameters (partial derivatives of model parameters), and is used to guide how to adjust the model parameters to minimize the total loss function. By calculating the gradient and then adjusting the model parameters based on the gradient, on the one hand, the difference between the target text and the sample label recognized by the speech recognition model can be kept within a small range, so that the speech recognition model can continuously process sequences of indefinite length (streaming speech) and avoid the situation where a 30-second pause is required when the probability of recognition is low. On the other hand, the speech recognition model can quickly learn the association between the mixed audio stream and the sample label, reduce the learning samples, and improve the learning efficiency.

[0272] In this embodiment, the forward probability Indicates that before time step t and sample label u, the sample audio features are observed and sample labels The probability of is defined as the forward probability. Optionally, the forward probability is predicted by the following formula:

[0273] ;

[0274] Backward Probability Indicates that after time step t and sample label u, the sample audio features are observed and sample labels The probability of is defined as the backward probability. Optionally, the backward probability is predicted by the following formula:

[0275] ;

[0276] In this embodiment, the total loss function is:

[0277] ;

[0278] Where X is the mixed audio stream, H1 is the first audio feature, H2 is the second audio feature, Y1 is the first sample label corresponding to the first audio feature, and Y2 is the second sample label corresponding to the second audio feature; Lrnnt is the RNNT loss function, which is the loss function between the input audio feature H and the corresponding sample label Y; Lheat is the loss value.

[0279] In this example, the total loss function is composed of Lrnnt (Y1, H1) + Lrnnt (Y2, H2), where H1 is equivalent to the first audio feature extracted from the sample mixed audio stream, H2 is equivalent to the second audio feature extracted from the sample audio stream, Y1 is the sample label obtained based on the first audio feature, Y2 is the sample label obtained based on the second audio feature, Lrnnt (Y1, H1) represents the loss value of the probability of identifying the first sample label based on the first audio feature, and Lrnnt (Y2, H2) represents the loss value of the probability of identifying the second sample label based on the second audio feature. For mixed audio streams, the total loss value is calculated based on the loss values ​​of different objects. On the one hand, it can ensure that the speech recognition model balances the learning efficiency of different objects during the learning process, so that the recognition accuracy of each object is balanced. On the other hand, the model is adjusted more carefully during the training process to reduce specific types of errors, with good scalability and high efficiency, avoiding considering all possible label arrangements, thereby reducing calculation and memory requirements.

[0280] Alternatively, Lrnnt can be calculated using the following formula:

[0281] ;

[0282] in, is the forward probability, is the backward probability, U is the length of the sample label, and u is the sample label corresponding to the current time step.

[0283] In this example, the loss between the input audio features and the sample labels is calculated by combining the forward probability and the backward probability, so that the loss tends to be updated during the learning process of the speech recognition model. In this way, the learned speech recognition model has a smaller recognition error during application, and can learn longer audio streams.

[0284] Based on the embodiments of the present application, the trained speech recognition model can realize timely recognition and response to mixed audio streams, avoiding the situation where a pause is required after recognizing an audio stream of a certain length.

[0285] It should be noted that in the embodiment of the present application, a basic model is pre-trained with limited sample labels (errors are allowed) to reduce training samples and data acquisition costs. The basic model has low computing power requirements and is easy to deploy on the end side. For example, when applied to a car, the speech recognition model is applied to the car's electronic equipment.

[0286] During the use of the speech recognition model, the model parameters of the speech recognition model are updated through heuristic error allocation rules. Specifically, refer to the update process described in the above embodiment. The model parameters of the speech recognition model are updated based on specific abnormal situations during use, so that the model is more suitable for speech recognition in application scenarios and the recognition accuracy is getting higher and higher.

[0287] The present application also provides a speech recognition device, including:

[0288] An extraction module, used to extract features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object;

[0289] A first prediction module, used to obtain a first feature vector of the speech emitted by the target object according to the first audio feature;

[0290] A second prediction module, used to predict a second feature vector of a speech that may be uttered by the target object according to the second audio feature;

[0291] The recognition processing module is used to recognize and obtain the target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

[0292] The speech recognition device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0293] The embodiment of the present invention also provides a training device for a speech recognition model, comprising:

[0294] An acquisition module is used to acquire training samples, where the training samples include a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream;

[0295] The speech recognition model is used to perform the following processing on the input sample mixed audio stream to identify the target text corresponding to the speech emitted by the target object in the sample mixed audio stream; perform feature extraction on the sample mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects other than the target object; obtain the first feature vector of the speech emitted by the target object based on the first audio feature, and predict the second feature vector of the speech that may be emitted by the target object based on the second audio feature; and identify the target text based on the first feature vector and the second feature vector;

[0296] The training module is used to calculate the loss value of the target text relative to the corresponding sample label based on the total loss function, and adjust the model parameters of the speech recognition model according to the loss value.

[0297] The speech recognition model training device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0298] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor;

[0299] The memory is used to store computer programs / instructions; the processor is used to execute the computer programs / instructions stored in the memory to implement the methods involved in the above embodiments.

[0300] The electronic device also includes a communication interface and a CAN bus, wherein the processor is used to provide computing power and control capabilities, and can be a GPU, CPU, NPU, MCU, FPGA, etc. The storage device includes an internal memory and a non-volatile memory. The non-volatile memory stores a computer program that implements the above method. The internal memory provides an environment for program startup and operation. The communication interface is used to communicate with an external terminal by wire or wireless.

[0301] In this embodiment, the electronic device may be an electronic component in a car, or may be a car.

[0302] The present invention also provides a computer-readable storage medium / computer program product, in which computer-readable storage medium stores computer execution instructions / the computer program product includes computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the methods involved in the above-mentioned embodiments.

[0303] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A speech recognition method, characterized in that: The method comprises: Extracting features from a mixed audio stream to obtain a first audio feature of a target object and a second audio feature of another object other than the target object, wherein the mixed audio stream includes speech of at least two objects, and the target object is an object distinguished from the at least two objects; Obtaining a first feature vector of the speech emitted by the target object according to the first audio feature; Predicting, based on the second audio feature, a second feature vector associated with the speech that may be uttered by the target object; A target text corresponding to the speech uttered by the target object in the mixed audio stream is identified based on the first feature vector and the second feature vector.

2. The identification method according to claim 1, characterized in that: The step of identifying a target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector includes: Performing time-synchronous fusion on the first feature vector and the second feature vector to obtain a fused feature vector; According to the fused feature vector, a target text corresponding to the speech uttered by the target object in the mixed audio stream is identified.

3. The identification method according to claim 1, characterized in that: The identification method further includes: Predicting a backward probability of a speech that the target object may utter at a next time step based on the first audio feature; The step of identifying a target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector includes: Based on the first feature vector, the second feature vector, the backward probability and the forward probability of the speech emitted by the target object predicted in the previous time step, the target text corresponding to the speech emitted by the target object in the mixed audio stream is obtained through preset conditional probability prediction.

4. The identification method according to claim 1, characterized in that: The step of extracting features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of another object other than the target object includes: Preprocessing the mixed audio stream to obtain mixed audio features; Predicting a mask of speech emitted by a target object based on the mixed audio features; Obtaining a first audio feature of the target object according to the mask and the mixed audio feature; According to the complement corresponding to the mask and the mixed audio feature, a second audio feature of an object other than the target object is obtained.

5. The identification method according to any one of claims 1 to 4, characterized in that: Applied to a computing device deployed with a speech recognition model, the speech recognition model includes a mask network, a long short-term memory network, a prediction network and a joint network, wherein: The mask network is used to extract features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object; The long short-term memory network is used to obtain a first feature vector of the speech emitted by the target object according to the first audio feature; The prediction network is used to predict, based on the second audio feature, a second feature vector associated with the speech that may be uttered by the target object; The joint network is used to identify the target text corresponding to the speech emitted by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

6. The identification method according to claim 5, characterized in that: After the recognition obtains the target text corresponding to the speech emitted by the target object in the mixed audio stream, the method further includes: When it is monitored that the target text corresponding to the speech uttered by the target object at the current moment is abnormal, the model parameters of the speech recognition model are optimized based on the abnormal scenario of the target text, the application field of speech recognition and the corrected text, wherein the corrected text is the correct text corresponding to the abnormal target text.

7. The identification method according to claim 6, characterized in that: The optimizing the model parameters of the speech recognition model based on the abnormal scenario of the target text, the application field of speech recognition and the correction text includes: Training the speech recognition model according to the abnormal scenario of the target text, the application field of speech recognition and the correction text, and adjusting the loss of the speech recognition model; adjusting forward probability algorithm model parameters and backward probability algorithm model parameters based on the adjusted loss; Predicting the forward probability of the speech that the target object may make at a moment before the current moment based on the adjusted forward probability algorithm model parameters, and predicting the backward probability of the speech that the target object may make at a moment after the current moment based on the adjusted backward probability algorithm model parameters; Calculating a gradient according to the forward probability, the backward probability, and the revised text; Model parameters of the speech recognition model are adjusted based on the gradient.

8. The identification method according to claim 6 or 7, characterized in that: The model parameters of the speech recognition model are optimized through the cloud.

9. A method for training a speech recognition model, characterized in that: include: Acquire a training sample, where the training sample includes a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream; The sample mixed audio stream is input into a speech recognition model, and the following processing is performed in the speech recognition model to identify and obtain a target text corresponding to the speech emitted by the target object in the sample mixed audio stream: feature extraction is performed on the sample mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects other than the target object; a first feature vector of the speech emitted by the target object is obtained based on the first audio feature, and a second feature vector associated with the speech that may be emitted by the target object is predicted based on the second audio feature; the target text is identified and obtained based on the first feature vector and the second feature vector; Calculate the loss value of the target text relative to the corresponding target sample label based on the total loss function; The model parameters of the speech recognition model are adjusted according to the loss value.

10. The training method according to claim 9, characterized in that: The speech recognition model includes a mask network, which performs feature extraction on the sample mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object, including: Inputting the sample mixed audio stream into the preprocessing encoder of the mask network, preprocessing the sample mixed audio stream based on the preprocessing encoder, and obtaining sample mixed audio features; Inputting the sample audio features into the mask encoder of the mask network, processing the sample audio features based on the mask encoder, and predicting a mask of the speech emitted by the target object; Obtaining a first audio feature of the target object according to the mask and the sample audio feature; According to the complement corresponding to the mask and the sample audio feature, a second audio feature of an object other than the target object is obtained.

11. The training method according to claim 9, characterized in that: The speech recognition model includes a long short-term memory network, and obtaining a first feature vector of the speech emitted by the target object according to the first audio feature includes: Inputting the first audio feature into the long short-term memory network, performing feature extraction on the first audio feature based on the long short-term memory network, and obtaining a first feature vector of the speech emitted by the target object; And / or, the speech recognition model includes a prediction network, and the predicting, based on the second audio feature, of a second feature vector of the speech that may be uttered by the target object includes: The second audio feature is input into the prediction network, and based on the prediction network, the speech that may be uttered by the target object corresponding to the second audio feature is predicted, and a corresponding second feature vector is generated.

12. The training method according to claim 9, characterized in that: The training method further comprises: Predicting a backward probability of a speech that the target object may utter at a next time step based on the first audio feature; The step of identifying the target text according to the first feature vector and the second feature vector includes: Based on the first feature vector, the second feature vector, the backward probability and the forward probability of the speech uttered by the target object predicted in the previous time step, the target text is obtained through preset conditional probability prediction.

13. The training method according to any one of claims 9 to 12, characterized in that: The speech recognition model further includes an audio encoder and a label encoder, and the training method further includes: Input the first audio feature into the audio encoder, encode the first audio feature based on the audio encoder, and encode the sample label according to the label encoder; Predicting a backward probability of a speech that the target object may utter at a next time step based on the first audio feature; Predicting the gradient of the loss function with respect to the model parameters according to the backward probability and the forward probability of the speech emitted by the target object predicted at the previous time step; The adjusting the model parameters of the speech recognition model according to the loss value includes: When the loss value is greater than or equal to a preset threshold, the model parameters are adjusted according to the gradient.

14. The training method according to claim 13, characterized in that: The training method further comprises: The total loss function is constructed based on the error between the target text and the target sample label predicted by the first audio feature and the error between the target text and the target sample label predicted by the second audio feature.

15. A speech recognition device, characterized in that: include: An extraction module, used to extract features from the mixed audio stream to obtain a first audio feature of a target object and a second audio feature of other objects other than the target object; A first prediction module, configured to obtain a first feature vector of the speech emitted by the target object according to the first audio feature; A second prediction module, configured to predict, based on the second audio feature, a second feature vector associated with a voice that may be uttered by the target object; The recognition processing module is used to recognize and obtain the target text corresponding to the speech uttered by the target object in the mixed audio stream according to the first feature vector and the second feature vector.

16. A training device for a speech recognition model, characterized in that: include: An acquisition module is used to acquire a training sample, wherein the training sample includes a sample mixed audio stream and a sample label corresponding to the sample mixed audio stream; A speech recognition model is used to perform the following processing on the input sample mixed audio stream to identify and obtain a target text corresponding to the speech emitted by the target object in the sample mixed audio stream; Performing feature extraction on the sample mixed audio stream to obtain a first audio feature of a target object and a second audio feature of another object other than the target object; obtaining a first feature vector of a speech emitted by the target object based on the first audio feature, and predicting a second feature vector associated with a speech that may be emitted by the target object based on the second audio feature; and recognizing the target text based on the first feature vector and the second feature vector; A training module is used to calculate the loss value of the target text relative to the corresponding sample label based on the total loss function, and adjust the model parameters of the speech recognition model according to the loss value.

17. An electronic device, characterized in that: The electronic device comprises: a memory and a processor; The memory is used to store computer programs / instructions; the processor is used to implement the method according to any one of claims 1 to 14 according to the computer programs / instructions stored in the memory.

18. A vehicle, characterized in that: include: Memory, processor; The memory is used to store computer programs / instructions; The processor is configured to implement the method according to any one of claims 1 to 8 according to the computer program / instructions stored in the memory.

19. A computer-readable storage medium / computer program product, characterized in that: The computer-readable storage medium stores a computer program / instruction, or the computer program product includes a computer program / instruction, and the computer program / instruction is used to implement the method according to any one of claims 1 to 14 when executed by a processor.

Images