Method and device for processing and model training of streaming speech, equipment and medium

Abstract

The present disclosure provides a streaming voice processing and model training method, device, equipment and medium, relates to the technical field of computers, and particularly relates to the technical field of voice translation, model training and cloud services. The specific implementation scheme is: obtaining voice feature information of a streaming voice segment, and encoding the voice feature information to generate a voice recognition encoding vector and a voice translation encoding vector respectively; generating voice recognition text of the streaming voice segment according to a decoding result of the voice recognition encoding vector, and generating voice translation text of the streaming voice segment according to a decoding result of the voice translation encoding vector. The present disclosure can generate voice recognition text and voice translation text of a streaming voice segment in parallel, thereby avoiding error accumulation caused by generating voice translation results based on voice recognition results, and ensuring the accuracy of voice translation text.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the fields of speech translation, model training and cloud services, and especially to a method, apparatus, device and medium for processing and training streaming speech. Background Technology

[0002] With the development of technology and the advancement of globalization, language has become a major barrier to communication between people from different countries and regions. Translation technology aims to translate between different languages, helping people understand different languages. Existing translation technologies mainly consist of two parts: speech recognition and speech translation.

[0003] Most translation systems in the industry currently adopt a cascading model, which first performs speech recognition to generate a speech recognition result, and then performs speech translation on the speech recognition result to generate a speech translation result. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and medium for parallel recognition and translation of streaming speech segments.

[0005] According to one aspect of this disclosure, a method for processing streaming speech is provided, comprising:

[0006] Acquire speech feature information of streaming speech segments and encode the speech feature information to generate speech recognition coding vector and speech translation coding vector respectively;

[0007] Based on the decoding result of the speech recognition encoding vector, the speech recognition text of the streaming speech segment is generated, and

[0008] Based on the decoding result of the speech translation encoding vector, the speech translation text of the streaming speech segment is generated.

[0009] According to another aspect of this disclosure, a model training method is provided, comprising:

[0010] Obtain sample feature information of sample speech segments;

[0011] The sample feature information is used to perform speech recognition using a speech recognition model to generate speech recognition sample text of the sample speech segment; and the sample feature information is used to perform speech translation using a speech translation model to generate speech translation sample text of the sample speech segment.

[0012] The speech recognition model is trained based on the speech recognition sample text, and the speech translation model is trained based on the speech translation sample text.

[0013] According to another aspect of this disclosure, a streaming speech processing apparatus is provided, comprising:

[0014] The encoding module is used to acquire speech feature information of streaming speech segments and encode the speech feature information to generate speech recognition encoding vector and speech translation encoding vector, respectively.

[0015] The text generation module is used to generate speech recognition text of the streaming speech segment based on the decoding result of the speech recognition encoding vector, and

[0016] Based on the decoding result of the speech translation encoding vector, the speech translation text of the streaming speech segment is generated.

[0017] According to another aspect of this disclosure, a model training apparatus is provided, comprising:

[0018] The feature acquisition module is used to acquire sample feature information of sample speech segments;

[0019] The sample text generation module is used to perform speech recognition on the sample feature information using a speech recognition model to generate speech recognition sample text of the sample speech segment, and to perform speech translation on the sample feature information using a speech translation model to generate speech translation sample text of the sample speech segment.

[0020] The training module is used to train the speech recognition model based on the speech recognition sample text and to train the speech translation model based on the speech translation sample text.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1A This is a schematic diagram of the structure of some existing cascaded mode translation systems disclosed in the embodiments of this disclosure;

[0024] Figure 1B This is a flowchart of some streaming speech processing methods disclosed in the embodiments of this disclosure;

[0025] Figure 2A This is a flowchart of some other streaming speech processing methods disclosed in the embodiments of this disclosure;

[0026] Figure 2BThese are schematic diagrams of the structures of some streaming speech processing systems disclosed in embodiments of this disclosure;

[0027] Figure 3 This is a flowchart of some model training methods disclosed in the embodiments of this disclosure;

[0028] Figure 4 This is a flowchart of some other model training methods disclosed according to embodiments of this disclosure;

[0029] Figure 5 This is a flowchart of some other model training methods disclosed according to embodiments of this disclosure;

[0030] Figure 6 This is a schematic diagram of the structure of some streaming speech processing apparatuses disclosed in the embodiments of this disclosure;

[0031] Figure 7 These are schematic diagrams of the structures of some model training devices disclosed in the embodiments of this disclosure;

[0032] Figure 8 This is a block diagram of an electronic device used to implement the streaming speech processing method and / or model training method disclosed in the embodiments of this disclosure. Detailed Implementation

[0033] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] Most translation systems in the industry currently use a cascading model. Figure 1A These are schematic diagrams illustrating the structures of some existing cascaded mode translation systems disclosed in embodiments of this disclosure, such as... Figure 1A As shown, existing cascaded translation systems typically include four modules: a voice acquisition module, a voice recognition module, a judgment module, and a voice translation module. The voice acquisition module is used to acquire the user's voice; the voice recognition module is used to perform voice recognition to generate voice-recognized text; the judgment module is used to determine whether the voice-recognized text needs to be translated; and the voice translation module is used to translate the voice-recognized text to generate voice-translated text.

[0035] It can be seen that the existing cascade mode translation system serially generates the speech recognition text and the speech translation text, that is, the process of speech translation depends on the speech recognition text. If there are recognition errors in the speech recognition text, when performing speech translation on the speech recognition text, incorrect speech translation text will be obtained based on the incorrect speech recognition text. It can be seen that the errors in the speech recognition process will accumulate to the speech translation process, resulting in a significant reduction in the accuracy of the speech translation text.

[0036] Figure 1B FIG. is a flowchart of some processing methods for streaming speech disclosed in an embodiment of the present disclosure. This embodiment is applicable to the situation of performing speech recognition and speech translation on streaming speech segments. The method of this embodiment can be executed by the processing device for streaming speech disclosed in the embodiment of the present disclosure. The device can be implemented by software and / or hardware and can be integrated in any electronic device with computing capabilities.

[0037] As Figure 1B shown, the processing method for streaming speech disclosed in this embodiment may include:

[0038] S101. Obtain the speech feature information of the streaming speech segment, and encode the speech feature information to generate a speech recognition coding vector and a speech translation coding vector respectively.

[0039] Among them, the streaming speech segment represents the speech prefix of the entire sentence of speech issued by the user. For example, assuming that the entire sentence of speech is "The weather is really nice today", the streaming speech segment includes but is not limited to "Today", "Today the", "Today the weather", "Today the weather is" and "Today the weather is really nice", etc. The speech feature information represents the feature vector obtained by performing speech feature extraction on the streaming speech segment, and is the audio feature of the streaming speech segment.

[0040] In one implementation, obtain the streaming speech segment issued by the user in real time, and perform speech feature extraction on the streaming speech segment through a feature extraction algorithm to obtain the speech feature information of the streaming speech segment. Among them, the feature extraction algorithm includes but is not limited to the Mel-frequency cepstral coefficient algorithm, the linear prediction coefficient algorithm, the linear prediction cepstral coefficient algorithm, the linear frequency algorithm, the discrete wavelet transform algorithm, and the convolutional neural network model algorithm, etc. This embodiment does not specifically limit the feature extraction algorithm.

[0041] The speech feature information is input into a first encoder after training to perform speech recognition encoding, thereby mapping the speech feature information into a hidden layer, i.e., a vector representation containing a natural language sequence, to obtain a speech recognition encoding vector. Similarly, the speech feature information is input into a second encoder after training to perform speech translation encoding, thereby mapping the speech feature information into a hidden layer, i.e., a vector representation containing a natural language sequence, to obtain a speech translation encoding vector. The first and second encoders have different encoder parameters. The encoder parameters of the first encoder are obtained by training the model based on the standard text of the speech recognition of sample speech segments and the speech recognition sample text obtained from recognizing the sample speech segments. The encoder parameters of the second encoder are obtained by training the model based on the standard text of the speech translation of sample speech segments and the speech translation sample text obtained from translating the sample speech segments.

[0042] By acquiring speech feature information from streaming speech segments and encoding this information, speech recognition encoding vectors and speech translation encoding vectors are generated, achieving parallel encoding of speech feature information and laying the data foundation for subsequent parallel decoding.

[0043] S102. Based on the decoding result of the speech recognition encoding vector, generate the speech recognition text of the streaming speech segment, and based on the decoding result of the speech translation encoding vector, generate the speech translation text of the streaming speech segment.

[0044] Speech recognition text means converting streaming speech segments into text in the same language. For example, if the lost speech segment is Chinese, the speech recognition text will also be in Chinese. Speech translation text means converting streaming speech segments into text in a different language. For example, if the lost speech segment is Chinese, the speech translation text can be in English, French, Japanese, etc.

[0045] In one implementation, a speech recognition encoding vector is input to a trained first decoder to perform speech recognition decoding, thereby mapping a vector representation containing a natural language sequence to a natural language sequence, generating speech recognition text of a streaming speech segment; and a speech translation encoding vector is input to a trained second decoder to perform speech translation decoding, thereby mapping a vector representation containing a natural language sequence to a natural language sequence, generating speech translation text of a streaming speech segment; wherein the first decoder and the second decoder have different decoder parameters. The decoder parameters of the first decoder are obtained by training a model based on the speech recognition standard text of the sample speech segment and the speech recognition sample text of the sample speech segment output by the first decoder. The decoder parameters of the second decoder are obtained by training a model based on the speech translation standard text of the sample speech segment and the speech translation sample text of the sample speech segment output by the second decoder.

[0046] This disclosure obtains speech feature information from streaming speech segments and encodes the speech feature information to generate speech recognition encoding vectors and speech translation encoding vectors, respectively. Based on the decoding result of the speech recognition encoding vectors, speech recognition text of the streaming speech segments is generated, and based on the decoding result of the speech translation encoding vectors, speech translation text of the streaming speech segments is generated. This enables the parallel generation of speech recognition text and speech translation text of streaming speech segments, avoiding the accumulation of errors caused by generating speech translation results based on speech recognition results in the prior art, and ensuring the accuracy of speech translation text.

[0047] Figure 2A This is a flowchart of some other streaming speech processing methods disclosed in the embodiments of this disclosure, which are further optimized and extended based on the above technical solutions, and can be combined with the above optional implementation methods.

[0048] like Figure 2A As shown, the streaming speech processing method disclosed in this embodiment may include:

[0049] S201. Obtain the speech feature information of the streaming speech segment.

[0050] In one implementation, a user-generated streaming speech segment is input into a trained convolutional neural network model, which then extracts features from the streaming speech segment and outputs the speech feature information of the streaming speech segment.

[0051] S202. The speech feature information is encoded by the speech recognition encoder in the speech recognition model to generate a speech recognition encoding vector, and the speech feature information is encoded by the speech translation encoder in the speech translation model to generate a speech translation encoding vector.

[0052] In one implementation, speech feature information is input into the speech recognition encoder of a trained speech recognition model, causing the speech recognition encoder to encode the speech feature information and output a speech recognition encoding vector. Furthermore, speech feature information is also input into the speech translation encoder of a trained speech translation model, causing the speech translation encoder to encode the speech feature information and output a speech translation encoding vector.

[0053] By encoding speech feature information using a speech recognition encoder in a speech recognition model to generate a speech recognition encoding vector, and by encoding speech feature information using a speech translation encoder in a speech translation model to generate a speech translation encoding vector, the effect of parallel encoding of speech feature information is achieved. This makes the generation of speech translation encoding vectors independent of speech recognition results, thus avoiding the problem of error accumulation.

[0054] S203. Decode the speech recognition encoding vector using the speech recognition decoder in the speech recognition model to generate a speech recognition decoding vector; generate speech recognition text of streaming speech segments based on the speech recognition decoding vector.

[0055] In one implementation, the speech recognition encoding vector output by the speech recognition encoder is input into the speech recognition decoder of the speech recognition model. The speech recognition decoder first decodes the speech recognition encoding vector through the decoding layer to generate a speech recognition decoding vector. Then, the speech recognition decoding vector is converted into a logits format vector through the linear layer in the speech recognition decoder. Finally, the logits format vector is converted into a probability value through the softmax layer in the speech recognition decoder. The speech recognition text is then determined from the dictionary corresponding to the speech recognition decoder based on the probability value.

[0056] The speech recognition decoder in the speech recognition model decodes the speech recognition encoding vector to generate a speech recognition decoding vector, and then generates the speech recognition text of the streaming speech segment based on the speech recognition decoding vector. This achieves the effect of speech recognition of streaming speech segments and meets the user's business needs for speech recognition.

[0057] S204. Decode the speech translation encoding vector using the speech translation decoder in the speech translation model to generate a speech translation decoding vector; generate the speech translation text of the streaming speech segment based on the speech translation decoding vector.

[0058] In one implementation, the speech translation encoding vector output by the speech translation encoder is input into the speech translation decoder of the speech translation model. The speech translation decoder first decodes the speech translation encoding vector through a decoding layer to generate a speech translation decoding vector. Then, the speech translation decoding vector is converted into a logits format vector through a linear layer in the speech translation decoder. Finally, the logits format vector is converted into a probability value through a softmax layer in the speech translation decoder. The speech translation text is then determined from the dictionary corresponding to the speech translation decoder based on the probability value.

[0059] The speech translation model decodes the speech translation encoding vector to generate a speech translation decoding vector, and then generates the speech translation text of the streaming speech segment based on the speech translation decoding vector. The speech translation effect of the streaming speech segment meets the user's business needs for speech translation. Furthermore, the generation of speech translation text does not rely on speech recognition text, avoiding the problem of error accumulation and ensuring the accuracy of speech translation text.

[0060] Based on the above embodiments, optionally, after S204, the method further includes:

[0061] Based on speech feature information, predict whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text; wherein, the whole sentence speech translation text is generated based on the whole sentence speech to which the streaming speech segment belongs; if so, the speech translation text is output.

[0062] In this context, the "speech-translated text" is a text prefix of the "complete speech-translated text," meaning that the speech-translated text is part of the complete speech-translated text, and vice versa. For example, if the speech-translated text is "what a wonderful," and the complete speech-translated text is "what a wonderful day," then the speech-translated text is a text prefix of the complete speech-translated text. Conversely, if the speech-translated text is "what a nice," and the complete speech-translated text is "what a wonderful day," then the speech-translated text is not a text prefix of the complete speech-translated text.

[0063] In one implementation, speech feature information is input into a trained prediction model. The model then predicts whether the translated text of a streaming speech segment is a text prefix of a complete translated text, based on the speech feature information, and outputs the prediction result. If the prediction result indicates that the translated text is a text prefix of a complete translated text, the translated text is output for the user to view. If the prediction result indicates that the translated text is not a text prefix of a complete translated text, the translated text is not output.

[0064] By predicting whether the speech translation text of a streaming speech segment is a text prefix of the whole sentence speech translation text based on speech feature information, the whole sentence speech translation text is generated based on the whole sentence speech to which the streaming speech segment belongs. If it is, the speech translation text is output, which ensures the accuracy of the output speech translation text and avoids the problem that the speech translation text of the streaming speech segment is not a text prefix of the whole sentence speech translation text, thus improving the readability of the translation results.

[0065] Based on the above embodiments, optionally, predicting whether the translated text of a streaming speech segment is a text prefix of the translated text of a whole sentence based on speech feature information includes:

[0066] The speech feature information is classified using a binary classification model, and the classification results are used to determine whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text.

[0067] In one implementation, a binary classification model is pre-trained based on the sample feature information of sample speech segments. After the speech feature information of the streaming speech segments is input into the trained binary classification model, the model outputs a binary classification result. Then, based on the binary classification result, it is determined whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text.

[0068] For example, the binary classification model outputs "0" or "1". When the output is "0", it means that the translated text is not a text prefix of the whole sentence translated text; when the output is "1", it means that the translated text is a text prefix of the whole sentence translated text. This embodiment only uses the binary classification result of "0" and "1" for explanation and does not limit the specific form of the binary classification result.

[0069] By classifying speech feature information using a binary classification model, and determining whether the speech translation text of a streaming speech segment is a text prefix of the whole sentence speech translation text based on the classification results, the accuracy of prediction is improved and the user experience is enhanced.

[0070] Based on the above embodiments, optionally, the speech recognition encoder and the speech translation encoder may adopt at least one of the following encoder structures:

[0071] Emformer encoder, Masked Transformer encoder, Scout Network encoder, unidirectional recurrent neural network encoder, and block-based encoder.

[0072] Since the speech translation and speech recognition text of the streaming speech segment are generated in real time, it is not necessary to obtain the entire sentence before speech translation and speech recognition. Therefore, in the speech recognition encoder and speech translation encoder of this embodiment, the encoding of the streaming speech segment at the current moment depends only on the encoding results of the previous moment, but not on the encoding results of the subsequent moment.

[0073] In one implementation, an encoder structure selected from the following—Emformer encoder, Masked Transformer encoder, Scout Network encoder, unidirectional recurrent neural network encoder, and block-based encoder—is used as the speech recognition encoder; and an encoder structure selected from the following—Emformer encoder, Masked Transformer encoder, Scout Network encoder, unidirectional recurrent neural network encoder, and block-based encoder—is used as the speech translation encoder. The encoder structures of the speech recognition encoder and the speech translation encoder can be the same, for example, both using a Masked Transformer encoder, or they can be different, for example, the speech recognition encoder uses a Masked Transformer encoder, and the speech translation encoder uses an Emformer encoder, etc. This is for illustrative purposes only and does not constitute a specific limitation on the encoder structures used in the speech recognition encoder and the speech translation encoder.

[0074] By setting the speech recognition encoder and speech translation encoder to adopt at least one of the following encoder structures: Emformer encoder, Masked Transformer encoder, Scout Network encoder, unidirectional recurrent neural network encoder, and block-based encoder, the speech recognition encoder and speech translation encoder can be adapted to streaming speech recognition and translation scenarios, ensuring that the method execution process can proceed smoothly.

[0075] Figure 2B These are schematic diagrams of the structures of some streaming speech processing systems disclosed in embodiments of this disclosure, such as... Figure 2B As shown, the streaming speech processing system includes a convolutional neural network model, a speech recognition model, a speech translation model, and a binary classification model. The speech recognition model includes a speech recognition encoder and a speech recognition decoder, and the speech translation model includes a speech translation encoder and a speech translation decoder.

[0076] A convolutional neural network model is used to extract features from streaming speech segments and output the speech feature information of the streaming speech segments. A speech recognition encoder encodes the speech feature information and outputs a speech recognition encoded vector; a speech recognition decoder decodes the speech recognition encoded vector to generate a speech recognition decoded vector, and generates the speech recognition text of the streaming speech segment based on the speech recognition decoded vector. A speech translation encoder encodes the speech feature information and outputs a speech translation encoded vector; a speech translation decoder decodes the speech translation encoded vector to generate a speech translation decoded vector, and generates the speech translation text of the streaming speech segment based on the speech translation decoded vector. A binary classification model is used to classify the speech feature information and, based on the classification result, determines whether the speech translation text of the streaming speech segment is a text prefix of the entire sentence's speech translation text; if so, the speech translation text is output.

[0077] Figure 3 This is a flowchart of some model training methods disclosed in embodiments of this disclosure. These embodiments are applicable to the joint training of speech recognition and speech translation models. The methods in this embodiment can be executed by the model training apparatus disclosed in these embodiments. The apparatus can be implemented in software and / or hardware and can be integrated into any electronic device with computing capabilities.

[0078] like Figure 3 As shown, the model training method disclosed in this embodiment may include:

[0079] S301. Obtain sample feature information of sample speech segments.

[0080] Among them, the sample audio segments are pre-collected audio recordings, such as the voices of each participant in a recitation competition or meeting recordings, which can be used as sample audio segments.

[0081] In one embodiment, a feature extraction algorithm is used to extract speech features from sample speech segments to obtain sample feature information of the sample speech segments. The feature extraction algorithm includes, but is not limited to, the Mel-frequency cepstral coefficient algorithm, the linear prediction coefficient algorithm, the linear prediction cepstral coefficient algorithm, the linear frequency algorithm, the discrete wavelet transform algorithm, and the convolutional neural network model algorithm, etc. This embodiment does not specifically limit the feature extraction algorithm.

[0082] By acquiring sample feature information from sample speech segments, a data foundation is laid for subsequent model training.

[0083] S302. Using a speech recognition model, perform speech recognition on the sample feature information to generate speech recognition sample text of the sample speech segment; and using a speech translation model, perform speech translation on the sample feature information to generate speech translation sample text of the sample speech segment.

[0084] In one implementation, sample feature information is input into a speech recognition model to be trained. The speech recognition model first encodes the sample feature information to generate a first sample encoding vector, then decodes the first sample encoding vector to generate a first sample decoding vector, and finally generates speech recognition sample text of the sample speech segment based on the first sample decoding vector. Similarly, sample feature information is input into a speech translation model to be trained. The speech translation model first encodes the sample feature information to generate a second sample encoding vector, then decodes the second sample encoding vector to generate a second sample decoding vector, and finally generates speech translation sample text of the sample speech segment based on the second sample decoding vector.

[0085] By using a speech recognition model to perform speech recognition on sample feature information, speech recognition sample text of sample speech segments is generated. Similarly, by using a speech translation model to perform speech translation on sample feature information, speech translation sample text of sample speech segments is generated. This process yields the predicted values ​​(speech recognition sample text and speech translation sample text) output by the speech recognition model and speech translation model to be trained, thus laying the data foundation for subsequent model training based on the predicted values.

[0086] S303. Train the speech recognition model based on speech recognition sample text, and train the speech translation model based on speech translation sample text.

[0087] In one implementation, standard text for speech recognition and standard text for speech translation, pre-annotated with sample speech segments, are determined. A first loss value is determined based on the speech recognition sample text output by the speech recognition model to be trained and the standard text, and the model parameters of the speech recognition model to be trained are trained based on the first loss value. Furthermore, a second loss value is determined based on the speech translation sample text output by the speech translation model to be trained and the standard text, and the model parameters of the speech translation model to be trained are trained based on the second loss value.

[0088] This disclosure obtains sample feature information of sample speech segments, performs speech recognition on the sample feature information using a speech recognition model to generate speech recognition sample text of the sample speech segments, and performs speech translation on the sample feature information using a speech translation model to generate speech translation sample text of the sample speech segments. Then, the speech recognition model is trained based on the speech recognition sample text, and the speech translation model is trained based on the speech translation sample text. This achieves the effect of jointly training the speech recognition model and the speech translation model, improving the efficiency of model training. Furthermore, when the trained speech translation model performs speech translation, it does not rely on the speech recognition results of the speech recognition model, thereby avoiding the accumulation of errors caused by generating speech translation results based on speech recognition results and ensuring the accuracy of speech translation.

[0089] Figure 4 This is a flowchart of some other model training methods disclosed in the embodiments of this disclosure, which are further optimized and extended based on the above technical solutions, and can be combined with the above optional implementation methods.

[0090] like Figure 4 As shown, the model training method disclosed in this embodiment may include:

[0091] S401. Obtain sample feature information of sample speech segments.

[0092] S402. The sample feature information is encoded by the speech recognition encoder in the speech recognition model to generate the first sample encoding vector.

[0093] In one implementation, sample feature information is input into the speech recognition encoder of the speech recognition model to be trained, so that the speech recognition encoder encodes the sample feature information and outputs a first sample encoding vector.

[0094] S403. The first sample encoding vector is decoded by the speech recognition decoder in the speech recognition model to generate the first sample decoding vector.

[0095] In one implementation, the first sample encoding vector output by the speech recognition encoder is input into the speech recognition decoder of the speech recognition model to be trained, so that the speech recognition decoder decodes the first sample encoding vector through the decoding layer to generate the first sample decoding vector.

[0096] S404. Generate speech recognition sample text of the sample speech segment based on the first sample decoding vector.

[0097] In one implementation, the first sample decoding vector is converted into a logits format vector by a linear layer in the speech recognition decoder, and finally the logits format vector is converted into a probability value by a softmax layer in the speech recognition decoder. The speech recognition sample text is then determined in the dictionary corresponding to the speech recognition decoder based on the probability value.

[0098] The speech recognition model encodes the sample feature information to generate a first sample encoding vector, and then decodes the first sample encoding vector to generate a first sample decoding vector. Based on the first sample decoding vector, the speech recognition sample text of the sample speech segment is generated. Thus, the speech recognition model with the "encoder-decoder" structure performs speech recognition, ensuring the accuracy and timeliness of speech recognition.

[0099] S405. Determine the speech recognition standard text of the sample speech segment, and determine the first loss value based on the speech recognition standard text and the speech recognition sample text.

[0100] In one implementation, a first loss value is calculated between the standard text for speech recognition and the sample text for speech recognition using a pre-defined loss function.

[0101] Optionally, a first loss value is determined based on the standard text for speech recognition and the sample text for speech recognition, including:

[0102] The first loss value is determined using the following loss function:

[0103] L1 = -logP(y1 = y T1 |x1;θ e1 ,θ d1 )

[0104] Where L1 represents the first loss value, y1 represents the speech recognition sample text generated by the speech recognition model, and y T1 Let x1 represent the standard text for speech recognition, x1 represent the sample feature information input to the speech recognition model, and θ represent the standard text for speech recognition. e1 θ represents the first encoder parameter of the speech recognition encoder. d1 This represents the first decoder parameter of the speech recognition decoder. P(y1=y T1 |x1;θ e1 ,θ d1 The meaning of ) is that when the sample feature information input to the speech recognition model is x1, the first encoder parameter of the speech recognition encoder is θ. e1 The first decoder parameter of the speech recognition decoder is θ d1 At that time, the probability value of the speech recognition sample text output by the speech recognition model being the standard speech recognition text is given.

[0105] The first loss value is determined by using the following loss function: L1 = -logP(y1 = y T1 |x1;θ e1 ,θ d1 Since the exponential loss function is very sensitive to outliers and noise, it is very suitable for speech recognition scenarios, thus improving the accuracy of calculating the first loss value.

[0106] S406. Update the first encoder parameters of the speech recognition encoder and the first decoder parameters of the speech recognition decoder based on the first loss value.

[0107] In one implementation, an optimization algorithm, such as gradient descent, is used to determine the optimal encoder parameters corresponding to the speech recognition encoder and the optimal decoder parameters corresponding to the speech recognition decoder when the first loss value is minimized. Then, the first encoder parameters of the speech recognition encoder are updated according to the optimal encoder parameters, and the first decoder parameters of the speech recognition decoder are updated according to the optimal decoder parameters.

[0108] By determining the standard text for speech recognition of sample speech segments, and determining the first loss value based on the standard text and the sample text, and updating the first encoder parameters of the speech recognition encoder and the first decoder parameters of the speech recognition decoder based on the first loss value, the system achieves the effect of backpropagation updating the first encoder parameters of the speech recognition encoder and the first decoder parameters of the speech recognition decoder based on the first loss value, thereby improving the efficiency of model training.

[0109] Figure 5 This is a flowchart of some other model training methods disclosed in the embodiments of this disclosure, which are further optimized and extended based on the above technical solutions, and can be combined with the above optional implementation methods.

[0110] like Figure 5 As shown, the model training method disclosed in this embodiment may include:

[0111] S501. Obtain sample feature information of sample speech segments.

[0112] S502. The sample feature information is encoded by the speech translation encoder in the speech translation model to generate the second sample encoding vector.

[0113] In one implementation, sample feature information is input into the speech translation encoder of the speech translation model to be trained, so that the speech translation encoder encodes the sample feature information and outputs a second sample encoding vector.

[0114] S503. The second sample encoding vector is decoded by the speech translation decoder in the speech translation model to generate the second sample decoding vector.

[0115] In one implementation, the second sample encoded vector output by the speech translation encoder is input into the speech translation decoder of the speech translation model to be trained, so that the speech translation decoder decodes the second sample encoded vector through the decoding layer to generate the second sample decoded vector.

[0116] S504. Generate speech translation sample text of the sample speech segment based on the second sample decoding vector.

[0117] In one implementation, the second sample decoding vector is converted into a logits format vector by a linear layer in the speech translation decoder, and finally the logits format vector is converted into a probability value by a softmax layer in the speech translation decoder. The speech translation sample text is then determined in the dictionary corresponding to the speech translation decoder based on the probability value.

[0118] The speech translation model encodes sample feature information using a speech encoder to generate a second sample encoded vector, and then decodes this second sample encoded vector using a speech decoder to generate a second sample decoded vector. Based on this second sample decoded vector, the model generates the speech translation sample text for the sample speech segment. This "encoder-decoder" structure ensures the accuracy and timeliness of speech translation. Furthermore, the trained speech translation model does not rely on the speech recognition results of the speech recognition model, thus avoiding the accumulation of errors caused by generating speech translation results based on speech recognition results and guaranteeing the accuracy of speech translation.

[0119] S505. Determine the standard text for speech translation of the sample speech segment, and determine the second loss value based on the standard text and the sample text.

[0120] In one implementation, a second loss value is calculated between the standard text for speech translation and the sample text for speech translation using a pre-defined loss function.

[0121] Optionally, a second loss value is determined based on the standard text for speech translation and the sample text for speech translation, including:

[0122] The second loss value is determined using the following loss function:

[0123] L2 = -logP(y2 = y T2 |x1;θ e2 ,θ d2 )

[0124] Where L2 represents the second loss value, y2 represents the speech translation sample text generated by the speech translation model, and y T2 Let x1 represent the standard text for speech translation, x1 represent the sample feature information input to the speech translation model, and θ represent the standard text for speech translation. e1 The second encoder parameter represents the speech translation encoder; θ d1 This represents the second decoder parameter of the speech translation decoder. P(y2=y T2 |x1;θ e2 ,θ d2 The meaning of ) is that when the sample feature information input to the speech translation model is x1, the second encoder parameter of the speech translation encoder is θ. e2 The second decoder parameter of the speech translation decoder is θ d2 At that time, the probability value of the speech translation sample text output by the speech translation model being the speech translation standard text is given.

[0125] The second loss value is determined by using the following loss function: L2 = -logP(y2 = y T2 |x1;θ e2 ,θ d2 Since the exponential loss function is very sensitive to outliers and noise, it is very suitable for speech translation scenarios, thus improving the accuracy of calculating the second loss value.

[0126] S506. Update the second encoder parameters of the speech translation encoder and the second decoder parameters of the speech translation decoder based on the second loss value.

[0127] In one implementation, an optimization algorithm, such as gradient descent, is used to determine the optimal encoder parameters corresponding to the speech translation encoder and the optimal decoder parameters corresponding to the speech translation decoder when the second loss value is minimized. Then, the second encoder parameters of the speech translation encoder are updated according to the optimal encoder parameters, and the second decoder parameters of the speech translation decoder are updated according to the optimal decoder parameters.

[0128] By determining the standard text for speech translation of sample speech segments, and determining the second loss value based on the standard text and the sample text, and updating the second encoder parameters of the speech translation encoder and the second decoder parameters of the speech translation decoder based on the second loss value, the system achieves the effect of backpropagation updating the second encoder parameters of the speech translation encoder and the second decoder parameters of the speech translation decoder based on the second loss value, thereby improving the efficiency of model training.

[0129] Based on the above embodiments, this embodiment also provides a model training method, including:

[0130] The speech translation model is used to translate the sample feature information into speech, generating speech translation sample text of the sample speech segment;

[0131] Based on whether the speech translation sample text is a text prefix of the whole sentence speech translation sample text, a standard label value for the sample feature information is generated; if yes, the standard label value is "1", and if no, the standard label value is "0"; where the whole sentence speech translation sample text is generated based on the whole sentence sample text to which the sample speech segment belongs;

[0132] The sample feature information is input into the binary classification model to be trained, and the predicted label value is output. The binary classification model to be trained is trained based on the third loss value between the predicted label value and the standard label value. The binary classification model can be used to predict whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text. If so, the speech translation text is output.

[0133] Optionally, the third loss value can be determined using the following loss function:

[0134]

[0135] Where L3 represents the third loss value, y3 represents the predicted label value of the sample feature information at time 1:t, and y T3 Let x[1:t] represent the standard label value of the sample feature information at time 1:t, and let θ represent the sample feature information at time 1:t input to the binary classification model to be trained. p This represents the classification parameters of the binary classification model to be trained. P(y3=y T3 |x[1:t];θ p The meaning of ) is that when the sample feature information at time 1:t is x[1:t], the classification parameter of the binary classification model is θ. p At time t, the predicted label value of the sample feature information at time 1:t is the probability value of the standard label value.

[0136] Figure 6 This is a schematic diagram of the structure of some streaming speech processing apparatuses disclosed in embodiments of this disclosure, which can be applied to situations involving speech recognition and speech translation of streaming speech segments. The apparatus of this embodiment can be implemented in software and / or hardware and can be integrated into any electronic device with computing capabilities.

[0137] like Figure 6 As shown, the streaming speech processing apparatus 60 disclosed in this embodiment may include an encoding module 61 and a text generation module 62, wherein:

[0138] The encoding module 61 is used to acquire speech feature information of streaming speech segments and encode the speech feature information to generate speech recognition encoding vector and speech translation encoding vector respectively.

[0139] The text generation module 62 is used to generate speech recognition text of streaming speech segments based on the decoding results of the speech recognition encoding vector, and

[0140] Based on the decoding results of the speech translation encoding vector, the speech translation text of the streaming speech segment is generated.

[0141] Optional, encoding module 61, specifically used for:

[0142] The speech feature information is encoded by the speech recognition encoder in the speech recognition model to generate a speech recognition encoding vector, and

[0143] The speech feature information is encoded by the speech translation encoder in the speech translation model to generate a speech translation encoding vector.

[0144] Optional, text generation module, specifically used for:

[0145] The speech recognition decoder in the speech recognition model decodes the speech recognition encoding vector to generate the speech recognition decoding vector.

[0146] The speech recognition text is generated from the speech recognition decoding vectors to generate the speech recognition text of the streaming speech segments.

[0147] Optional, the text generation module is also used for:

[0148] The speech translation encoding vector is decoded by the speech translation decoder in the speech translation model to generate the speech translation decoding vector;

[0149] The speech translation text of the streaming speech segment is generated based on the speech translation decoding vector.

[0150] Optionally, the device also includes a text output module, specifically used for:

[0151] Based on speech feature information, predict whether the speech translation text of a streaming speech segment is a text prefix of the whole sentence speech translation text; wherein, the whole sentence speech translation text is generated based on the whole sentence speech to which the streaming speech segment belongs;

[0152] If so, the translated text will be output.

[0153] Optional, the text output module is also used for:

[0154] The speech feature information is classified using a binary classification model, and the classification results are used to determine whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text.

[0155] Optionally, the speech recognition encoder and the speech translation encoder may employ at least one of the following encoder structures:

[0156] Emformer encoder, Masked Transformer encoder, Scout Network encoder, unidirectional recurrent neural network encoder, and block-based encoder.

[0157] The streaming speech processing apparatus 60 disclosed in this embodiment can execute the streaming speech processing method disclosed in this embodiment, and has the corresponding functional modules and beneficial effects for executing the method. Content not described in detail in this embodiment can be referred to the description in the method embodiments of this disclosure.

[0158] Figure 7 This is a schematic diagram of the structure of some model training devices disclosed in embodiments of this disclosure, which can be applied to the joint training of speech recognition models and speech translation models. The device in this embodiment can be implemented in software and / or hardware and can be integrated into any electronic device with computing capabilities.

[0159] like Figure 7 As shown, the model training device 70 disclosed in this embodiment may include a feature acquisition module 71, a sample text generation module 72, and a training module 73, wherein:

[0160] The feature acquisition module 71 is used to acquire sample feature information of sample speech segments;

[0161] The sample text generation module 72 is used to perform speech recognition on the sample feature information through a speech recognition model to generate speech recognition sample text of the sample speech segment, and to perform speech translation on the sample feature information through a speech translation model to generate speech translation sample text of the sample speech segment.

[0162] The training module 73 is used to train the speech recognition model based on the speech recognition sample text and to train the speech translation model based on the speech translation sample text.

[0163] Optionally, the sample text generation module 72 is specifically used for:

[0164] The sample feature information is encoded by the speech recognition encoder in the speech recognition model to generate a first sample encoding vector;

[0165] The first sample encoding vector is decoded by the speech recognition decoder in the speech recognition model to generate the first sample decoding vector;

[0166] The speech recognition sample text of the sample speech segment is generated based on the first sample decoding vector.

[0167] Optionally, the sample text generation module 72 is further configured to:

[0168] The sample feature information is encoded by the speech translation encoder in the speech translation model to generate a second sample encoding vector;

[0169] The second sample encoding vector is decoded by the speech translation decoder in the speech translation model to generate the second sample decoding vector;

[0170] The speech translation sample text of the sample speech segment is generated based on the second sample decoding vector.

[0171] Optionally, the training module 73 is specifically used for:

[0172] Determine the speech recognition standard text of the sample speech segment, and determine a first loss value based on the speech recognition standard text and the speech recognition sample text;

[0173] The first encoder parameters of the speech recognition encoder and the first decoder parameters of the speech recognition decoder are updated based on the first loss value.

[0174] Optionally, the training module 73 is further used for:

[0175] The first loss value is determined using the following loss function:

[0176] L1 = -logP(y1 = y T1 |x1;θ e1 ,θ d1 )

[0177] Where L1 represents the first loss value, y1 represents the speech recognition sample text generated by the speech recognition model, and y T1 Let x1 represent the standard text for speech recognition, x1 represent the sample feature information input to the speech recognition model, and θ represent the standard text for speech recognition. e1 The first encoder parameter θ represents the speech recognition encoder. d1 This represents the first decoder parameter of the speech recognition decoder.

[0178] Optionally, the training module 73 is further used for:

[0179] Determine the standard text for speech translation of the sample speech segment, and determine a second loss value based on the standard text for speech translation and the sample text for speech translation;

[0180] The second encoder parameters of the speech translation encoder and the second decoder parameters of the speech translation decoder are updated based on the second loss value.

[0181] Optionally, the training module 73 is further used for:

[0182] The second loss value is determined using the following loss function:

[0183] L2 = -log P(y2 = y T2 |x1;θ e2 ,θ d2 )

[0184] Where L2 represents the second loss value, y2 represents the speech translation sample text generated by the speech translation model, and y T2 Let x1 represent the standard text for speech translation, x1 represent the sample feature information input to the speech translation model, and θ represent the standard text for speech translation. e2 The second encoder parameter θ represents the speech translation encoder. d2 This represents the second decoder parameter of the speech translation decoder.

[0185] The model training apparatus 70 disclosed in this embodiment can execute the model training method disclosed in this embodiment, and has the corresponding functional modules and beneficial effects for executing the method. Content not described in detail in this embodiment can be referred to the description in the method embodiments of this disclosure.

[0186] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0187] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0188] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0189] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0190] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0191] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as streaming speech processing methods and / or model training methods. For example, in some embodiments, the streaming speech processing methods and / or model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the streaming speech processing methods and / or model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform streaming speech processing methods and / or model training methods.

[0192] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0194] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0196] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0197] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0198] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0199] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for processing streaming speech, comprising: Acquire speech feature information of streaming speech segments and encode the speech feature information to generate speech recognition coding vector and speech translation coding vector respectively; Based on the decoding result of the speech recognition encoding vector, the speech recognition text of the streaming speech segment is generated, and based on the decoding result of the speech translation encoding vector, the speech translation text of the streaming speech segment is generated. Based on the speech feature information, predict whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text; wherein, the whole sentence speech translation text is generated based on the whole sentence speech to which the streaming speech segment belongs; the speech translation text of the streaming speech segment being a text prefix of the whole sentence speech translation text indicates that the speech translation text of the streaming speech segment is part of the whole sentence speech translation text; If yes, then the translated text of the streaming speech segment will be output; otherwise, the translated text of the streaming speech segment will not be output.

2. The method according to claim 1, wherein, The process of encoding the speech feature information to generate speech recognition encoding vectors and speech translation encoding vectors includes: The speech feature information is encoded by a speech recognition encoder in the speech recognition model to generate a speech recognition encoding vector. The speech feature information is encoded by the speech translation encoder in the speech translation model to generate a speech translation encoding vector.

3. The method according to claim 1 or 2, wherein, The step of generating the speech recognition text of the streaming speech segment based on the decoding result of the speech recognition encoding vector includes: The speech recognition encoded vector is decoded by the speech recognition decoder in the speech recognition model to generate a speech recognition decoded vector; The speech recognition text of the streaming speech segment is generated based on the speech recognition decoding vector.

4. The method according to any one of claims 1-2, wherein, The step of generating the speech-translated text of the streaming speech segment based on the decoding result of the speech translation encoding vector includes: The speech translation encoding vector is decoded by the speech translation decoder in the speech translation model to generate a speech translation decoding vector; The speech translation text of the streaming speech segment is generated based on the speech translation decoding vector.

5. The method according to claim 1, wherein, The step of predicting whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text based on the speech feature information includes: The speech feature information is classified using a binary classification model, and the speech translation text of the streaming speech segment is determined based on the classification results to determine whether it is a text prefix of the whole sentence speech translation text.

6. The method according to claim 2, wherein, The speech recognition encoder and the speech translation encoder adopt at least one of the following encoder structures: Emformer encoder, Masked Transformer encoder, Scout Network encoder, unidirectional recurrent neural network encoder, and block-based encoder.

7. A model training method, comprising: Obtain sample feature information of sample speech segments; The sample feature information is used to perform speech recognition using a speech recognition model to generate speech recognition sample text of the sample speech segment; and the sample feature information is used to perform speech translation using a speech translation model to generate speech translation sample text of the sample speech segment. The speech recognition model is trained based on the speech recognition sample text, and the speech translation model is trained based on the speech translation sample text; The method further includes: The sample feature information is translated into speech using a speech translation model to generate speech translation sample text of the sample speech segment; Based on whether the speech translation sample text is a text prefix of the whole sentence speech translation sample text, a standard label value for the sample feature information is generated; wherein, the whole sentence speech translation sample text is generated based on the whole sentence speech to which the sample speech segment belongs; The sample feature information is input into the binary classification model to be trained, and the predicted label value is output. The binary classification model to be trained is trained based on the third loss value between the predicted label value and the standard label value. The binary classification model is used to predict whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text. If so, the speech translation text of the streaming speech segment is output.

8. The method according to claim 7, wherein, The step of performing speech recognition on the sample feature information using a speech recognition model to generate speech recognition sample text of the sample speech segment includes: The sample feature information is encoded by the speech recognition encoder in the speech recognition model to generate a first sample encoding vector; The first sample encoding vector is decoded by the speech recognition decoder in the speech recognition model to generate the first sample decoding vector; The speech recognition sample text of the sample speech segment is generated based on the first sample decoding vector.

9. The method according to claim 7, wherein, The step of translating the sample feature information into speech using a speech translation model to generate speech translation sample text for the sample speech segment includes: The sample feature information is encoded by the speech translation encoder in the speech translation model to generate a second sample encoding vector; The second sample encoding vector is decoded by the speech translation decoder in the speech translation model to generate the second sample decoding vector; The speech translation sample text of the sample speech segment is generated based on the second sample decoding vector.

10. The method according to claim 8, wherein, The step of training the speech recognition model based on the speech recognition sample text includes: Determine the speech recognition standard text of the sample speech segment, and determine a first loss value based on the speech recognition standard text and the speech recognition sample text; The first encoder parameters of the speech recognition encoder and the first decoder parameters of the speech recognition decoder are updated based on the first loss value.

11. The method according to claim 10, wherein, Determining the first loss value based on the speech recognition standard text and the speech recognition sample text includes: The first loss value is determined using the following loss function: ； in, This represents the first loss value. This represents the speech recognition sample text generated by the speech recognition model. This represents the standard text for speech recognition. This represents the sample feature information input to the speech recognition model. This represents the first encoder parameter of the speech recognition encoder; This represents the first decoder parameter of the speech recognition decoder.

12. The method according to claim 9, wherein, The step of training the speech translation model based on the speech translation sample text includes: Determine the standard text for speech translation of the sample speech segment, and determine a second loss value based on the standard text for speech translation and the sample text for speech translation; The second encoder parameters of the speech translation encoder and the second decoder parameters of the speech translation decoder are updated based on the second loss value.

13. The method according to claim 12, wherein, The step of determining the second loss value based on the speech translation standard text and the speech translation sample text includes: The second loss value is determined using the following loss function: ； in, This represents the second loss value. This refers to the speech translation sample text generated by the speech translation model. This refers to the standard text for speech translation. This represents the sample feature information input to the speech translation model. This represents the second encoder parameter of the speech translation encoder; This represents the second decoder parameter of the speech translation decoder.

14. A streaming speech processing apparatus, comprising: The encoding module is used to acquire speech feature information of streaming speech segments and encode the speech feature information to generate speech recognition encoding vector and speech translation encoding vector, respectively. The text generation module is used to generate speech recognition text of the streaming speech segment based on the decoding result of the speech recognition encoding vector, and to generate speech translation text of the streaming speech segment based on the decoding result of the speech translation encoding vector. The text output module is specifically used for: Based on the speech feature information, predict whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text; wherein, the whole sentence speech translation text is generated based on the whole sentence speech to which the streaming speech segment belongs; the speech translation text of the streaming speech segment being a text prefix of the whole sentence speech translation text indicates that the speech translation text of the streaming speech segment is part of the whole sentence speech translation text; If yes, then the translated text of the streaming speech segment will be output; otherwise, the translated text of the streaming speech segment will not be output.

15. The apparatus according to claim 14, wherein, The encoding module is specifically used for: The speech feature information is encoded by a speech recognition encoder in the speech recognition model to generate a speech recognition encoding vector. The speech feature information is encoded by the speech translation encoder in the speech translation model to generate a speech translation encoding vector.

16. The apparatus according to claim 14 or 15, wherein, The text generation module is specifically used for: The speech recognition encoded vector is decoded by the speech recognition decoder in the speech recognition model to generate a speech recognition decoded vector; The speech recognition text of the streaming speech segment is generated based on the speech recognition decoding vector.

17. The apparatus according to any one of claims 14-15, wherein, The text generation module is further used for: The speech translation encoding vector is decoded by the speech translation decoder in the speech translation model to generate a speech translation decoding vector; The speech translation text of the streaming speech segment is generated based on the speech translation decoding vector.

18. The apparatus according to claim 14, wherein, The text output module is further used for: The speech feature information is classified using a binary classification model, and the speech translation text of the streaming speech segment is determined based on the classification results to determine whether it is a text prefix of the whole sentence speech translation text.

19. The apparatus according to claim 15, wherein, The speech recognition encoder and the speech translation encoder adopt at least one of the following encoder structures: Emformer encoder, Masked Transformer encoder, Scout Network encoder, unidirectional recurrent neural network encoder, and block-based encoder.

20. A model training device, comprising: The feature acquisition module is used to acquire sample feature information of sample speech segments; The sample text generation module is used to perform speech recognition on the sample feature information using a speech recognition model to generate speech recognition sample text of the sample speech segment, and to perform speech translation on the sample feature information using a speech translation model to generate speech translation sample text of the sample speech segment. The training module is used to train the speech recognition model based on the speech recognition sample text and to train the speech translation model based on the speech translation sample text. The training module is further configured to: The sample feature information is translated into speech using a speech translation model to generate speech translation sample text of the sample speech segment; Based on whether the speech translation sample text is a text prefix of the whole sentence speech translation sample text, a standard label value for the sample feature information is generated; wherein, the whole sentence speech translation sample text is generated based on the whole sentence speech to which the sample speech segment belongs; The sample feature information is input into the binary classification model to be trained, and the predicted label value is output. The binary classification model to be trained is trained based on the third loss value between the predicted label value and the standard label value. The binary classification model is used to predict whether the speech translation text of the streaming speech segment is a text prefix of the whole sentence speech translation text. If so, the speech translation text of the streaming speech segment is output.

21. The apparatus according to claim 20, wherein, The sample text generation module is specifically used for: The sample feature information is encoded by the speech recognition encoder in the speech recognition model to generate a first sample encoding vector; The first sample encoding vector is decoded by the speech recognition decoder in the speech recognition model to generate the first sample decoding vector; The speech recognition sample text of the sample speech segment is generated based on the first sample decoding vector.

22. The apparatus according to claim 20, wherein, The sample text generation module is further used for: The sample feature information is encoded by the speech translation encoder in the speech translation model to generate a second sample encoding vector; The second sample encoding vector is decoded by the speech translation decoder in the speech translation model to generate the second sample decoding vector; The speech translation sample text of the sample speech segment is generated based on the second sample decoding vector.

23. The apparatus according to claim 21, wherein, The training module is specifically used for: Determine the speech recognition standard text of the sample speech segment, and determine a first loss value based on the speech recognition standard text and the speech recognition sample text; The first encoder parameters of the speech recognition encoder and the first decoder parameters of the speech recognition decoder are updated based on the first loss value.

24. The apparatus according to claim 23, wherein, The training module is further used for: The first loss value is determined using the following loss function: ； in, This represents the first loss value. This represents the speech recognition sample text generated by the speech recognition model. This represents the standard text for speech recognition. This represents the sample feature information input to the speech recognition model. This represents the first encoder parameter of the speech recognition encoder; This represents the first decoder parameter of the speech recognition decoder.

25. The apparatus according to claim 22, wherein, The training module is further used for: Determine the standard text for speech translation of the sample speech segment, and determine a second loss value based on the standard text for speech translation and the sample text for speech translation; The second encoder parameters of the speech translation encoder and the second decoder parameters of the speech translation decoder are updated based on the second loss value.

26. The apparatus according to claim 25, wherein, The training module is further used for: The second loss value is determined using the following loss function: ； in, This represents the second loss value. This refers to the speech translation sample text generated by the speech translation model. This refers to the standard text for speech translation. This represents the sample feature information input to the speech translation model. This represents the second encoder parameter of the speech translation encoder; This represents the second decoder parameter of the speech translation decoder.

27. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-6 and / or claims 7-13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-6 and / or 7-13.

29. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6 and / or 7-13.

Images