Model training method, voice processing method and related equipment

Abstract

The invention provides a model training method, a voice processing method and related equipment. The method comprises the following steps: acquiring a sample audio and a text annotation and an audio annotation corresponding to the sample audio; performing first training processing on the language processing model according to the sample audio and the audio annotation to obtain a first audio processing model; and performing second training processing on the first audio processing model according to the sample audio and the text label to obtain a target audio processing model. According to the embodiment of the invention, the stability and accuracy of a model processing result can be improved.

Description

Model training methods, speech processing methods and related equipment Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method, a speech processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the widespread application of voice interaction technology, in order to improve audio processing performance, there are related solutions based on large language models (LLM) for audio processing. In this type of solution, pre-trained large models are generally used directly to process audio.

[0003] In related technologies, this method often suffers from poor stability and inaccuracy in the model processing results when processing audio. Summary of the Invention

[0004] This disclosure provides a model training method, a speech processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] Firstly, this disclosure provides a model training method, which includes:

[0006] Obtain sample audio and corresponding text and audio annotations, wherein the audio annotations are used to represent the audio features of the sample audio in a discretized form;

[0007] The language processing model is trained based on the sample audio and the audio annotation to obtain a first audio processing model, wherein the first training process is used to enable the first audio processing model to process audio coding unit sequences.

[0008] The first audio processing model is subjected to a second training process based on the sample audio and the text annotation to obtain a target audio processing model, wherein the second training process is used to enable the target audio processing model to have the ability to process audio.

[0009] Secondly, this disclosure provides a speech processing method, which includes:

[0010] Acquire the speech to be processed;

[0011] The speech to be processed is input into the target audio processing model to obtain the speech processing result of the speech to be processed; wherein the target audio processing model is obtained based on the model training method described in the first aspect.

[0012] Thirdly, this disclosure provides a model training apparatus, which includes:

[0013] The acquisition module is used to acquire sample audio and the corresponding text and audio annotations of the sample audio, wherein the audio annotations are used to represent the audio features of the sample audio in a discretized form;

[0014] The first training module is used to perform a first training process on the language processing model based on the sample audio and the audio annotation to obtain a first audio processing model, wherein the first training process is used to enable the first audio processing model to have the ability to process audio coding unit sequences.

[0015] The second training module is used to perform a second training process on the first audio processing model based on the sample audio and the text annotation to obtain a target audio processing model, wherein the second training process is used to enable the target audio processing model to have the ability to process audio.

[0016] Fourthly, this disclosure provides a speech processing apparatus, which includes:

[0017] The voice acquisition module is used to acquire the voice to be processed.

[0018] The speech processing module is used to input the speech to be processed into a target audio processing model to obtain the speech processing result of the speech to be processed; wherein the target audio processing model is obtained based on the model training method described in the first aspect.

[0019] Fifthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method or speech processing method described above.

[0020] Sixthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described model training method or speech processing method.

[0021] In a seventh aspect, this disclosure provides a computer program product that includes computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the model training method or speech processing method described above.

[0022] The embodiments provided in this disclosure first train a language processing model using sample audio and its discretized audio annotations to obtain a first audio processing model capable of processing audio coding unit sequences. This allows for the reconstruction and prediction of audio modalities based on the first audio processing model. Subsequently, the first audio processing model is second trained using sample audio and its text annotations, enabling the model to learn the deep semantic correspondence between audio and text. This achieves effective alignment of the audio and text semantic spaces, avoiding semantic errors that may result from simple feature splicing or shallow feature fusion. As a result, the trained target audio processing model can stably and accurately process audio input, improving the stability and accuracy of the model's processing results.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0025] Figure 1 is a flowchart of a model training method provided in an embodiment of this disclosure;

[0026] Figure 2 is a schematic diagram of obtaining the audio coding unit sequence provided in an embodiment of this disclosure;

[0027] Figure 3 is a first flowchart of the first training process provided in an embodiment of this disclosure;

[0028] Figure 4 is a second flowchart of the first training process provided in an embodiment of this disclosure;

[0029] Figure 5 is a schematic diagram of the first training process provided in an embodiment of this disclosure;

[0030] Figure 6 is a flowchart of the second training process provided in an embodiment of this disclosure;

[0031] Figure 7 is a flowchart of the third training process provided in an embodiment of this disclosure;

[0032] Figure 8 is a schematic diagram of the third training process provided in an embodiment of this disclosure;

[0033] Figure 9 is a flowchart of a speech processing method provided in an embodiment of this disclosure;

[0034] Figure 10 is a block diagram of a model training device provided in an embodiment of this disclosure;

[0035] Figure 11 is a block diagram of a voice processing device provided in an embodiment of the present disclosure;

[0036] Figure 12 is a block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0037] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0039] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0041] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0042] Traditional audio processing models typically use an acoustic model as the initial model, training it with high-quality and diverse training data to improve its audio processing capabilities. However, this training method requires significant effort in constructing training samples, making it time-consuming and labor-intensive. Therefore, as mentioned in the background section, considering that large models have already achieved strong language understanding capabilities through pre-training on large amounts of text, related technologies include schemes that use large models for audio data processing to reduce the rigid dependence on large-scale audio annotation and improve the model's generalization ability to unseen samples.

[0043] In the process of implementing this application, the applicant discovered that if audio data is directly input into a large model and the large model is trained, since the large model is often a model for text processing trained on a large amount of text annotation, even if a large amount of audio data is used to process the large model, the training process usually causes the model to rigidly convert audio features into text semantics, which may result in unstable model processing results and low accuracy.

[0044] To address this, the applicant discovered that during training, an audio-to-text model and a modal adaptation layer could be added before the main model. The audio-to-text model would identify the audio text, and the modal adaptation layer would transform the audio text into the same space as the embedding layer of the main model, obtaining its mapped features. These mapped features, along with the text embedding features, would then be simply concatenated and input into the main model for speech prediction. While this method can improve the model's audio processing capabilities to some extent, because it merely concatenates audio and text features and injects them into the main model, the main model has not yet stably learned how to process audio features, resulting in inconsistent results and low accuracy.

[0045] In view of this, embodiments of the present disclosure provide a model training method, comprising: acquiring sample audio and corresponding text and audio annotations, wherein the audio annotations are used to represent the audio features of the sample audio in a discretized form; performing a first training process on a language processing model based on the sample audio and audio annotations to obtain a first audio processing model, wherein the first training process is used to enable the first audio processing model to process audio coding unit sequences; and performing a second training process on the first audio processing model based on the sample audio and text annotations to obtain a target audio processing model, wherein the second training process is used to enable the target audio processing model to process audio.

[0046] Based on the model training method provided in this embodiment, a first audio processing model is obtained by using sample audio and its discretized audio annotations for first training, thereby enabling the processing of audio coding unit sequences. This allows for the reconstruction and prediction of audio modalities based on the first audio processing model. Subsequently, a second training process is performed on the first audio processing model based on sample audio and its text annotations, enabling the model to learn the deep semantic correspondence between audio and text. This achieves effective alignment of the audio and text semantic spaces, avoiding semantic errors that may result from simple feature splicing or shallow feature fusion. The trained target audio processing model can then stably and accurately process audio input, improving the stability and accuracy of the model's processing results.

[0047] The model training method and speech processing method according to embodiments of this disclosure can be executed by electronic devices such as terminal devices or servers. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0048] Figure 1 is a flowchart of a model training method provided in an embodiment of this disclosure. Referring to Figure 1, the method includes:

[0049] In step S11, the sample audio and the corresponding text and audio annotations are obtained.

[0050] Audio annotation is used to represent the audio features of sample audio in a discretized form.

[0051] In this embodiment, the sample audio can be the raw audio data used to train the model, and it can exist in the form of a signal or a file. The sample audio can be of fixed duration, for example, 30 seconds. Of course, audio shorter than 30 seconds can be padded with silence to reach 30 seconds. This audio can be, for example, a user's speech signal or an audio signal acquired and played by another device. It should be noted that when the sample audio is user speech, it is used with the user's authorization.

[0052] This text annotation can be the audio text corresponding to the sample audio. For example, it could be "The weather is nice today".

[0053] The audio annotation can be a representation obtained by discretizing the audio features of the sample audio. For example, the sample audio can be first input into an audio encoder for encoding processing to obtain its audio features, and then the discrete audio coding unit sequence obtained by vector quantization (VQ) of these audio features can be used as the audio annotation of the sample audio. The audio annotation can be in the form of [12,345,67,201,…].

[0054] It should be noted that the audio encoder can be any encoder used for feature encoding of audio data. In this embodiment, the audio encoder can be represented by Whister-VQ, which is used to obtain discrete audio coding units for audio. The Whister can be composed of the Whisper Encoder and vector quantizer in the Whisper model. Specifically, the features encoded by the encoder used for feature encoding in the Whisper model, i.e., one of the last or last few layers of the Whisper Encoder, can be used as the audio features of the sample audio, and vector quantization can be performed to obtain its audio annotation. The Whister model is an open-source audio recognition model that can be used to convert audio into text; of course, the audio encoder can also be obtained based on encoders in other speech recognition models and other quantizers used for vector quantization, without special limitations here.

[0055] In step S12, the language processing model is trained based on the sample audio and audio annotations to obtain the first audio processing model.

[0056] The first training process is used to enable the first audio processing model to process audio coding unit sequences.

[0057] The language processing model can be an initially obtained pre-trained large language model, such as a text processing model based on the Transformer structure, or a text processing model with other structures; or it can be a model obtained by adjusting part of the structure of the initial language processing model, such as a model obtained by replacing the text output head of the language processing model with an audio output head, without any special limitation here.

[0058] In this embodiment of the disclosure, an audio encoding unit refers to an audio token, that is, a basic unit obtained after discretizing audio features. For example, “12” in the above audio annotation [12,345,67,201,…] is an audio token, which is a discrete symbol that the model can process.

[0059] In step S13, the first audio processing model is subjected to a second training process based on the sample audio and text annotations to obtain the target audio processing model.

[0060] The second training process is used to enable the target audio processing model to process audio.

[0061] For example, the sample audio can be a user's speech signal, such as the user saying "How are you?". This audio signal can exist in the form of an audio file or an audio stream. The corresponding text annotation is "How are you?", and its audio annotation can be the audio token sequence [12,345,67,201] obtained by inputting the sample audio into Whisper-VQ. Then, the sample audio and audio annotation can be input into the LLM model for the first stage of training. During the training process, the model learns how to process the audio coding unit sequence [12,345,67,201] and predict its subsequent speech, thereby establishing a basic understanding and prediction ability of audio modalities, enabling the first audio processing model to process audio coding unit sequences. Then, the first audio processing model is trained in the second stage using the sample audio and its corresponding text annotation, enabling the model to learn how to map audio content to the corresponding text semantics, achieving audio-text alignment, thereby obtaining the final target audio processing model. This target audio model can directly process audio input and output the corresponding text content, possessing complete audio processing capabilities.

[0062] The above describes how to perform the first and second training processes on a language processing model using a sample audio and its corresponding text and audio annotations. It can be understood that in the actual training process, the training set may include multiple training data in the form of {sample audio (audio-question), text annotation (text-answer), audio annotation (audio-token)}. This training process may include multiple rounds to finally obtain a language processing model that meets the convergence condition.

[0063] As can be seen, according to the embodiments of this disclosure, by using sample audio and its discretized audio annotations to perform a first training process on the language processing model, a first audio processing model with the ability to process audio coding unit sequences is obtained, thereby enabling the reconstruction and prediction of audio modalities based on the first audio processing model; then, by performing a second training process on the first audio processing model based on sample audio and its text annotations, the model can learn the deep semantic correspondence between audio and text, thereby achieving effective alignment of the audio and text semantic spaces, avoiding semantic errors that may be caused by simple feature splicing or shallow feature fusion, and enabling the trained target audio processing model to process audio input stably and accurately, improving the stability and accuracy of the model processing results.

[0064] In some embodiments, audio annotation may include an audio coding unit sequence of sample audio; the audio coding unit sequence is obtained by encoding the sample audio and quantizing it in a preset number of dimensions.

[0065] Specifically, the audio coding unit sequence of the sample audio can be obtained by performing feature encoding processing on the sample audio to obtain its audio features, and then quantizing the audio features.

[0066] The preset quantity can be 4096 or 8192, or it can be set as needed. In the following explanation, we will take the preset quantity of 4096 as an example, that is, quantizing the audio into a 4096-dimensional audio space for illustration.

[0067] Please refer to Figure 2, which is a schematic diagram of obtaining the audio coding unit sequence provided in the embodiments of this disclosure. For ease of understanding, the following description, in conjunction with Figure 2, illustrates the acquisition of the audio coding unit sequence of input audio based on the Whisper-VQ described above. As shown in Figure 2, the input audio can be input into the block causal self-attention mechanism and feed-forward network of the Whisper model for feature encoding. The resulting audio features are then pooled by a pooling layer and input into a vector quantizer for vector quantization, thus obtaining the audio coding unit sequence that represents the audio features of the input audio in a discretized form. It should be noted that in actual implementation, the audio features used to extract the input audio and the vector quantizer used for vector quantization can be set as needed. Figure 2 is only for illustrative purposes, and this disclosure does not impose any special limitations on this.

[0068] Please refer to Figure 3, which is a first flowchart of the first training process provided in an embodiment of this disclosure. As shown in Figure 3, in some embodiments, the language processing model may include an embedding layer and an audio output head; the number of embedding vectors in the embedding layer is set to a preset number, and the embedding layer is used to encode and generate embedding vector representations that characterize semantic features; the audio output head is used to perform audio coding unit sequence prediction processing on the semantic feature vectors. In step S12, the first training process of the language processing model based on sample audio and audio annotations to obtain the first audio processing model may include the following step S31.

[0069] In step S31, the embedding layer and audio output head of the language processing model are subjected to a first training process based on the sample audio and audio coding unit sequence to obtain the first audio processing model.

[0070] This audio output head, namely llm-audio-head, can be used to map the semantic feature vectors obtained from the internal processing of the model to the audio coding space for audio coding unit sequence prediction processing.

[0071] That is, by using sample audio and sample coding unit sequences to train the embedding layer of the language processing model and the audio output head for predicting audio coding unit sequences of semantic feature vectors, the model learns how to process and predict audio coding unit sequences. This enables the stable and accurate injection of audio modalities into the model, thereby establishing a basic understanding and prediction capability of audio modalities. This allows the first audio processing model obtained from the first training process to have the ability to accurately process audio coding unit sequences.

[0072] In some embodiments, the method may further include: replacing the text output header of the initial language processing model with an audio output header to obtain the language processing model; wherein the text output header is used to perform text encoding unit sequence prediction processing on the semantic feature vector.

[0073] This initial language processing model can be a pre-trained language processing model.

[0074] Specifically, in traditional LLM models, the model is typically used for text processing, i.e., natural language processing. After receiving input text, the model usually encodes the input text based on an embedding layer. In this embedding layer, word tokens or character tokens in the text are mapped to corresponding embedding vector representations based on the text vocabulary space learned during pre-training. These vectors are continuously updated during model training to capture information such as semantic relationships, syntactic structure, and contextual dependencies between words. Through the embedding layer, discrete text tokens are transformed into continuous semantic feature vectors, which can then be further processed by subsequent attention mechanism layers and fully connected layers.

[0075] In this embodiment of the disclosure, considering that when the embedding layer in the initial language processing model maps the audio input to an embedding vector representation, if it still uses the text vocabulary to represent the audio encoding unit, i.e., the audio token, as an embedding vector, the audio token will still be rigidly mapped to the text vocabulary space. Since the semantics represented by the audio token are often inconsistent with the semantics represented by the text token and there is a significant difference, and the dimensions of the audio encoding space do not match those of the text vocabulary space, if this is still done, although it can improve the stability and accuracy of the model processing results to a certain extent compared with related technologies, the model processing results are still not stable and accurate enough.

[0076] Therefore, in this embodiment of the present disclosure, before the first training process, the number of embedding vectors used in the embedding layer of the initial language processing model can be synchronously set to the preset number, such as 4096, based on the audio space quantization dimension used when encoding and quantizing the sample audio into audio tokens, for example, 4096 dimensions, and these 4096 embedding vectors are randomly initialized. This setting ensures that the number of embedding vectors in the embedding layer matches the quantization dimension of the audio space, guaranteeing that during training, the model can accurately learn the mapping relationship between audio tokens and corresponding semantic feature vectors, thereby stably and accurately injecting the audio modality into the language processing model. This enables the language processing model to accurately process audio token sequences, further improving the stability and accuracy of the model's processing results.

[0077] Similarly, since the text output head in the initial language processing model, namely the LLM-text-head, is designed for the text vocabulary space, its output dimension corresponds to the number of text tokens, and it is often not directly used to predict audio coding units.

[0078] Therefore, in this embodiment of the present disclosure, before the first training process, the text output header in the initial language processing model can be replaced with an audio output header and randomly initialized. This allows the language processing model to learn how to accurately map the semantic feature vectors of the sample audio to the corresponding audio coding space, rather than to the text vocabulary space, during the training process. This enables accurate prediction of audio token sequences, allowing the model to correctly learn the distribution patterns and sequence relationships of audio coding units, thereby improving the stability and accuracy of the model's processing results.

[0079] Please refer to Figures 4 and 5, which are respectively a flowchart and a schematic diagram of the second training process provided in the embodiments of this disclosure. As shown in Figures 4 and 5, in some embodiments, the sample audio may include a first audio segment and a second audio segment, where the first audio segment corresponds to a first subsequence of the audio coding unit sequence, and the second audio segment corresponds to a second subsequence of the audio coding unit sequence.

[0080] In step S31 above, the language processing model is trained first based on the sample audio and the audio coding unit sequence to obtain the first audio processing model, which may include steps S41-S43, which will be described in detail below.

[0081] In step S41, the first subsequence is input into the language processing model to obtain the first predicted sequence corresponding to the second audio segment.

[0082] In step S42, the first loss value between the second subsequence and the first predicted sequence is obtained.

[0083] In step S43, the parameters of the embedding layer and the audio output head are adjusted based on the first loss value to obtain the first audio processing model.

[0084] As shown in Figure 5, in this embodiment of the present disclosure, in order to enable the trained model to process audio coding unit sequences, the sample audio can be divided into two audio segments during the first training process. For example, it can be set as a first audio segment consisting of a preceding part, as shown in Figure 5, and a second audio segment consisting of a following part. Correspondingly, the audio coding units of the sample audio can also be divided into a first subsequence and a second subsequence corresponding to the two audio segments, respectively. For example, when the sample audio is 30 seconds long, the first audio segment can be the first 15 seconds of audio, and the second audio segment can be the last 15 seconds of audio.

[0085] Of course, in actual implementation, the first audio segment and the second audio segment can be set as needed according to the prediction mechanism of the pre-trained language processing model used. For example, the first audio segment can be a 15-second audio segment in any middle part of the sample audio, and the second audio segment can be an audio segment outside of the 15-second segment. This embodiment does not make any special limitation on this.

[0086] Additionally, it should be noted that, for ease of understanding, Figure 5 uses the audio coding unit sequence of sample audio obtained based on Whisper-VQ as an illustration. It can be understood that during the first training process, the parameters of Whisper-VQ remain unchanged. Furthermore, during the first training process, it is also possible to first obtain the audio coding unit sequence of each sample audio in the training set offline based on Whisper-VQ. Thus, during the first training process, the audio coding unit sequence of each sample audio can be directly used for training without the need for real-time acquisition.

[0087] As shown in Figure 5, during the first training process, the first subsequence of the first audio segment can be input into the language processing model. The language processing model can predict the audio token of the second audio segment based on the audio token in the first subsequence, and obtain the first predicted sequence by prompting "Please predict the audio in the next 15 seconds" (not shown in the figure). Then, by obtaining the first loss value representing the error between the first predicted sequence and the second subsequence, the parameters of the embedding layer and the audio output head of the language processing model can be adjusted to convergence based on the first loss value. The first loss value can be obtained based on any loss function algorithm, such as cross-entropy loss, without any special limitation here.

[0088] As can be seen, based on the embodiments of this disclosure, by dividing the sample audio into a first audio segment and a second audio segment, and using the corresponding first and second subsequences for prediction training, the model can learn the sequence dependencies between audio coding units during the first training process. At the same time, by adjusting the parameters of the embedding layer and audio output head of the language processing model based on the first loss value, the obtained first audio processing model can more accurately predict subsequent audio coding unit sequences, thereby improving the model's ability to model audio coding unit sequences and its prediction stability, so as to stably inject the audio modality into the language processing model and lay the foundation for realizing audio-text alignment training during the subsequent second training process.

[0089] In some embodiments, in step S43, adjusting the parameters of the embedding layer and the audio output head based on the first loss value to obtain a first audio processing model may include: adjusting the parameters of the embedding layer and the audio output head of the embedded language processing model, as well as the low-rank adaptation LoRA module embedded in the embedded language processing model, based on the first loss value to obtain the first audio processing model; wherein the embedded language processing model is obtained by embedding a LoRA module in the attention mechanism layer of the language processing model.

[0090] That is, in some embodiments, when adjusting the parameters of the embedding layer and audio output head of the language processing model based on the first loss value, given the large number of parameters of the language processing model, in order to improve the training speed, it is not necessary to adjust all the parameters of the language processing model. Instead, the low-rank adaptation (LoRA) technique can be used to fine-tune the model.

[0091] Specifically, a LoRA module can be embedded in the Transformer layer of the language processing model to obtain an embedded language processing model. During the first training process, the parameters of the embedding layer, audio output head, and LoRA module of the embedded language processing model can be adjusted according to the first loss value, while other pre-trained parameters of the language processing model, such as the parameters of the original attention layer and feedforward network layer of the Transformer, can remain fixed during the first training process.

[0092] Based on this training method, while ensuring training efficiency, the model can quickly adapt to the processing task of audio coding unit sequences, resulting in a first audio processing model that meets the first preset convergence condition. It can be understood that the first preset convergence condition can be a first loss value less than a first preset threshold, which can be set as needed.

[0093] Please refer to Figure 6, which is a flowchart of the second training process provided in an embodiment of this disclosure. As shown in Figure 6, in some embodiments, after obtaining the first audio processing model, in step S13, the first audio processing model is subjected to a second training process based on sample audio and text annotations to obtain the target audio processing model, which may include the following steps S61-63.

[0094] In step S61, the sample audio is input into the second audio processing model to obtain the predicted and recognized text of the sample audio.

[0095] The second audio processing model includes a text output header, which is used to perform text encoding unit sequence prediction processing on the semantic feature vector; the second audio processing model is obtained based on the first audio processing model.

[0096] The text output head can be the original text output head of the initial language model, i.e., the pre-trained language processing model, i.e., llm-text-head. It is an output module used to map the semantic feature vectors obtained from the internal processing of the model to the text vocabulary space.

[0097] That is, in some embodiments, the first audio processing model may include an audio output head for performing audio coding unit sequence prediction processing on semantic feature vectors; the method may also include replacing the audio output head of the first audio processing model with a text output head to obtain a second audio processing model.

[0098] The predicted recognized text refers to the text predicted by the second audio processing model after performing audio recognition processing on the sample audio, which corresponds to the audio content of the sample audio.

[0099] In step S62, a second loss value is obtained between the text annotation and the predicted recognized text.

[0100] The second loss value can be a loss value used to represent the error between the predicted recognized text and the real text annotation of the sample audio. For example, it can be the cross-entropy loss, which can be used to reflect the accuracy of the model for audio recognition results.

[0101] In step S63, the parameters of the text output head of the second audio processing model are adjusted based on the second loss value to obtain the target audio processing model.

[0102] Specifically, since the first audio processing model obtained through the first training process is configured with an audio output head, i.e., llm-audio-head, in step S13, during the second training process, although the first audio processing model can be directly trained based on sample audio and text annotations to obtain a target audio processing model capable of processing audio, since the audio output head is mainly used for audio coding unit sequence prediction processing and is not trained with a large amount of natural text, the target audio processing model obtained in this way, although capable of stable and accurate processing of audio input, may lack cross-modal processing capabilities.

[0103] Therefore, in order to enable the final trained target audio processing model to learn stable and accurate cross-modal audio processing capabilities, before the second training process, the audio output head in the language processing model of the first training stage can be replaced with the original text output head. This allows the model to map the semantic feature vector obtained after processing the audio tokens internally to the text vocabulary space during the second training period, so as to output the text recognition result that can accurately represent the audio content of the sample audio.

[0104] Specifically, during the second training process of the second audio processing model, sample audio can be input into the second audio processing model. During this training period, the model can obtain the embedding vector representation of the sample audio through the embedding layer adapted to the audio modality, and then extract semantic feature vectors through the Transformer layer and LoRA module, and finally output the predicted recognition text of the sample audio through the text output head (llm-text-head).

[0105] Then, by obtaining the second loss value between the predicted and recognized text and the text annotation of the sample audio, and fine-tuning the second audio processing model based on the second loss value, the model can learn the ability to process audio across modalities, while also improving the stability and accuracy of the target audio model's processing results.

[0106] As can be seen, based on the embodiments of this disclosure, after obtaining the first audio processing model through the first training process, the original text output head of the initial language processing model can be reused, allowing the model to reuse the original text semantic mapping capability without redesigning the output module adapted to the text task, thus ensuring the compatibility and stability of the model's text output. At the same time, it can further establish a deep semantic association between audio and text, avoiding the cross-modal semantic misalignment problem caused by simply splicing or shallowly fusing audio features and text features in related technologies, thereby improving the model's generalization ability and the stability and accuracy of the audio processing results.

[0107] In addition, in some embodiments, the second training process of the first audio processing model based on the sample audio and text annotations to obtain the target audio processing model in step S13 may include: a third training process of the first audio processing model based on the noisy sample audio and audio annotations corresponding to the sample audio to obtain a third audio processing model; wherein, the third training process is used to enable the third audio processing model to have the ability to remove audio noise; the noisy sample audio is obtained by adding audio noise to the sample audio; and a fourth training process of the third audio processing model based on the sample audio and text annotations to obtain the target audio processing model.

[0108] Specifically, noisy sample audio can be obtained by adding audio noise from different sources and with different signal-to-noise ratios to the sample audio. This audio noise can be, for example, environmental noise, white noise, and reverberation with different signal-to-noise ratios, to simulate audio input in a real scene.

[0109] In related technologies, when training a model with audio processing capabilities based on a language processing model, the training set is often constructed by only containing clean audio, ignoring the model's robustness in real noisy environments. This often leads to a sharp drop in model performance in complex acoustic scenarios.

[0110] Therefore, in this embodiment of the present disclosure, in order to improve the robustness of the target audio processing model obtained by the final training, during the second training process, for the first audio processing model obtained by the first training process, a noisy sample audio can be obtained by adding audio noise to the sample audio. Based on the noisy sample audio and audio annotation, the first audio processing model is subjected to a third training process, so that the trained third audio processing model can learn the ability to reconstruct a clean audio coding unit sequence based on the noisy audio coding unit sequence, that is, it has the ability to remove audio noise in the input audio. Afterwards, a fourth training process can be performed on the third audio processing model according to the sample audio and text annotation to obtain the target audio processing model.

[0111] It should be noted that the fourth training process for the third audio processing model based on the sample audio and text annotations can include: inputting the sample audio into the fourth audio processing model to obtain the predicted recognition text of the sample audio; the fourth audio processing model includes a text output head, which is used to perform text encoding unit sequence prediction processing on the semantic feature vector; the fourth audio processing model is obtained based on the third audio processing model; obtaining the fourth loss value between the text annotations and the predicted recognition text of the sample audio; adjusting the parameters of the text output head of the fourth audio processing model based on the fourth loss value to obtain the target audio processing model. The third audio processing model includes an audio output head, which is used to perform audio encoding unit sequence prediction processing on the semantic feature vector; this method can include: replacing the audio output head of the third audio processing model with a text output head to obtain the fourth audio processing model. The detailed processing procedure can be found in steps S61-S63 above, and will not be repeated here.

[0112] Please refer to Figures 7 and 8, which are respectively a flowchart and a schematic diagram of the third training process provided in this embodiment. As shown in Figures 7 and 8, in this embodiment, the audio annotation includes the audio coding unit sequence of the sample audio; the noisy sample audio includes a third audio segment and a fourth audio segment, the fourth audio segment corresponding to the third subsequence of the clean audio coding unit sequence. For ease of understanding, Figure 8 directly illustrates the clean audio coding unit sequence of the sample audio, which does not contain audio noise. It can be understood that in actual implementation, the clean audio coding unit sequence can be obtained by inputting the sample audio into Whisper-VQ as shown in Figure 8. In addition, considering that in the third training process, the audio token sequence corresponding to the third audio segment of the noisy sample audio is usually input into the model for training, Figure 8 directly illustrates the process of inputting the third audio segment into Whisper-VQ to obtain its corresponding noisy audio coding unit sequence.

[0113] In this embodiment, as shown in Figure 7, the third audio processing model is obtained by performing a third training process on the first audio processing model based on the noisy sample audio and audio annotations, which may include the following steps S71-S74.

[0114] In step S71, the noisy frequency coding unit sequence of the third audio segment is obtained.

[0115] In step S72, the noisy frequency coding unit sequence is input into the first audio processing model to obtain the second prediction sequence corresponding to the fourth audio segment.

[0116] In step S73, a third loss value is obtained between the third subsequence and the second predicted sequence.

[0117] In step S74, the parameters of the first audio processing model are adjusted based on the third loss value to obtain the third audio processing model.

[0118] Specifically, as shown in Figure 8, in order to enable the model to learn the ability to reconstruct clean audio, the noisy sample audio can be divided into a third audio segment and a fourth audio segment. The third audio segment can be, for example, the first 15 seconds of the noisy sample audio, and the fourth audio segment can be the last 15 seconds of the noisy sample audio. Then, the noisy frequency coding unit sequence of the third audio segment can be obtained based on Whisper-VQ, that is, the noisy frequency token sequence of the first 15 seconds can be obtained. The noisy frequency token sequence is then input into the first audio processing model so that the model can be trained to learn the ability to remove noise from the noisy frequency token sequence and predict the clean audio token sequence.

[0119] In addition, the third loss value can be a numerical value that characterizes the error between the third subsequence and the second predicted sequence. For example, it can be obtained by calculating the cross-entropy loss between the two, without any special limitation here.

[0120] It should be noted that during the adjustment of the parameters of the first audio processing model based on the third loss value, the main parameters of the first audio processing model can also be kept unchanged, and only the parameters of its embedding layer, audio output head and embedded LoRA module can be adjusted. The detailed processing process will not be described here.

[0121] As can be seen, based on the embodiments of this disclosure, the model can learn the ability to reconstruct clean audio through the third training process described above. Then, by performing a fourth training process on the obtained third audio processing model, the robustness of the audio processing results of the final trained target audio processing model can be improved.

[0122] As can be seen, based on the training method provided in this embodiment, before the first training process, a pre-trained initial language processing model with sufficient representation and generation capabilities can be structurally adjusted, for example, by adjusting its embedding layer and replacing its text output header with an audio output header to obtain the language processing model corresponding to the initial language processing model; the language processing model is then subjected to the first training process, thereby enabling the first audio processing model obtained from the first training process to learn the ability to process audio coding unit sequences and perform audio coding unit sequence modeling by stably injecting audio modal features into the language processing model; subsequently, during the second training process, the first audio processing model can be trained to improve its denoising capabilities, enabling the obtained third audio processing model to reconstruct clean audio in a real noisy environment; furthermore, the third audio processing model can be subjected to a fourth training process using sample audio for audio recognition processing, thereby enabling the finally trained target audio processing model to have cross-modal audio processing capabilities and improving the stability, robustness, and accuracy of the model for audio processing results.

[0123] Understandably, since the language processing model during the first training process includes an audio output head, and this audio output head can be replaced with the original text output head during the second training process, the model can map the semantic feature vector obtained after processing the audio tokens internally to the text vocabulary space. Therefore, in the final target audio processing model, the audio-to-text capability can be extended to downstream tasks of training audio-to-audio, such as end-to-end voice interaction and voice translation.

[0124] Please refer to Figure 9, which is a flowchart of a voice processing method provided in an embodiment of this disclosure. This method can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server. As shown in Figure 9, the voice processing method may include the following steps S91-S92.

[0125] In step S91, the speech to be processed is acquired.

[0126] In step S92, the speech to be processed is input into the target audio processing model to obtain the speech processing result.

[0127] The target audio processing model is obtained based on the model training method of the present disclosure.

[0128] The voice to be processed can be a voice signal or a voice file obtained with the user's authorization; no special restrictions are imposed here.

[0129] After obtaining the speech to be processed in step S91, the speech to be processed is input into the target audio processing model finally trained based on the above model training method. The target audio processing model performs speech processing on the speech to be processed based on the corresponding input prompt words to obtain the corresponding speech processing result.

[0130] It is understood that the voice processing result will vary depending on the input prompt word. In some embodiments, the voice processing result may include at least one of the following: the recognized text corresponding to the voice to be processed, the response text corresponding to the voice to be processed, the response voice corresponding to the voice to be processed, the translated text in a first preset language corresponding to the voice to be processed, and the translated voice in a second preset language corresponding to the voice to be processed.

[0131] The first and second preset languages ​​can be set as needed, for example, English and Cantonese respectively.

[0132] That is, when the input prompt words for the speech to be processed are used to prompt the model to perform tasks such as audio text recognition, audio text response, audio response, and audio translation, the model can output the corresponding speech processing results.

[0133] As can be seen, the speech processing method provided in this disclosure improves the accuracy of the speech processing results by performing speech processing on the speech to be processed based on a target audio processing model with stable injected audio modality and cross-modal audio processing capabilities.

[0134] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0135] In addition, this disclosure also provides a model training and speech processing apparatus, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the model training methods and speech processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0136] Figure 10 is a block diagram of a model training device provided in an embodiment of this disclosure.

[0137] Referring to FIG10, an embodiment of the present disclosure provides a model training device, which may include: an acquisition module 101, a first training module 102, and a second training module 103.

[0138] The acquisition module 101 is used to acquire sample audio and the corresponding text annotation and audio annotation, wherein the audio annotation is used to represent the audio features of the sample audio in a discretized form.

[0139] The first training module 102 is used to perform a first training process on the language processing model based on the sample audio and the audio annotation to obtain a first audio processing model, wherein the first training process is used to enable the first audio processing model to have the ability to process audio coding unit sequences.

[0140] The second training module 103 is used to perform a second training process on the first audio processing model based on the sample audio and the text annotation to obtain a target audio processing model, wherein the second training process is used to enable the target audio processing model to have the ability to process audio.

[0141] In some embodiments, the audio annotation includes an audio coding unit sequence of the sample audio; the audio coding unit sequence is obtained by encoding the sample audio and quantizing it in a preset number of dimensions; the language processing model includes an embedding layer and an audio output head; the number of embedding vectors in the embedding layer is set to the preset number, and the embedding layer is used to encode and generate embedding vector representations that characterize semantic features; the audio output head is used to perform audio coding unit sequence prediction processing on the semantic feature vectors; when the first training module 102 performs a first training process on the language processing model based on the sample audio and the audio annotation to obtain a first audio processing model, it can be used to: perform the first training process on the embedding layer and the audio output head of the language processing model based on the sample audio and the audio coding unit sequence to obtain the first audio processing model.

[0142] In some embodiments, the apparatus further includes a first adjustment module, which can be used to: replace the text output header of the initial language processing model with the audio output header to obtain the language processing model; wherein the text output header is used to perform text encoding unit sequence prediction processing on the semantic feature vector.

[0143] In some embodiments, the sample audio includes a first audio segment and a second audio segment, the first audio segment corresponding to a first subsequence of the audio coding unit sequence, and the second audio segment corresponding to a second subsequence of the audio coding unit sequence; when the first training module 102 performs the first training process on the embedding layer and the audio output head of the language processing model according to the sample audio and the audio coding unit sequence to obtain the first audio processing model, it can be used to: input the first subsequence into the language processing model to obtain a first prediction sequence corresponding to the second audio segment; obtain a first loss value between the second subsequence and the first prediction sequence; and adjust the parameters of the embedding layer and the audio output head based on the first loss value to obtain the first audio processing model.

[0144] In some embodiments, when the first training module 102 performs a second training process on the first audio processing model based on the sample audio and the text annotation to obtain a target audio processing model, it can be used to: input the sample audio into the second audio processing model to obtain the predicted recognition text of the sample audio; the second audio processing model includes a text output head, which is used to perform text encoding unit sequence prediction processing on the semantic feature vector; the second audio processing model is obtained based on the first audio processing model; obtain a second loss value between the text annotation and the predicted recognition text; adjust the parameters of the text output head of the first audio processing model based on the second loss value to obtain the target audio processing model.

[0145] In some embodiments, the first audio processing model includes an audio output head for performing audio coding unit sequence prediction processing on semantic feature vectors; the device further includes a second adjustment module, which can be used to replace the audio output head of the first audio processing model with the text output head to obtain the second audio processing model.

[0146] In some embodiments, when the second training module 103 performs a second training process on the first audio processing model based on the sample audio and the text annotation to obtain a target audio processing model, it can be used to: perform a third training process on the first audio processing model based on the noisy sample audio corresponding to the sample audio and the audio annotation to obtain a third audio processing model; wherein, the third training process is used to enable the third audio processing model to have the ability to remove audio noise; the noisy sample audio is obtained by adding audio noise to the sample audio; and a fourth training process is performed on the third audio processing model based on the sample audio and the text annotation to obtain the target audio processing model.

[0147] In some embodiments, the audio annotation includes the audio coding unit sequence of the sample audio; the noisy sample audio includes a third audio segment and a fourth audio segment, the fourth audio segment corresponding to a third sub-sequence of the audio coding unit sequence; when the second training module 103 performs a third training process on the first audio processing model based on the noisy sample audio corresponding to the sample audio and the audio annotation to obtain a third audio processing model, it can be used to: obtain the noisy audio coding unit sequence of the third audio segment; input the noisy audio coding unit sequence into the first audio processing model to obtain a second prediction sequence corresponding to the fourth audio segment; obtain a third loss value between the third sub-sequence and the second prediction sequence; adjust the parameters of the first audio processing model based on the third loss value to obtain the third audio processing model.

[0148] Figure 11 is a block diagram of a voice processing device provided in an embodiment of this disclosure.

[0149] Referring to FIG11, an embodiment of the present disclosure provides a voice processing device, which may include: a voice acquisition module 201 and a voice processing module 202.

[0150] The voice acquisition module 201 is used to acquire the voice to be processed.

[0151] The speech processing module 202 is used to input the speech to be processed into the language processing model to obtain the speech processing result of the speech to be processed; wherein the language processing model is obtained based on the above-mentioned model training method.

[0152] Figure 12 is a block diagram of an electronic device provided in an embodiment of this disclosure.

[0153] Referring to FIG12, an embodiment of the present disclosure provides an electronic device, which includes: at least one processor 701; at least one memory 702; and one or more I / O interfaces 703 connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by at least one processor 701, and the one or more computer programs are executed by at least one processor 701 to enable at least one processor 701 to perform the above-described model training method and speech processing method.

[0154] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned model training method and speech processing method. The computer-readable storage medium may be volatile or non-volatile.

[0155] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described model training method and speech processing method.

[0156] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0157] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0158] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0159] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0160] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0161] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0162] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0163] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0165] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A model training method, characterized in that, include: Obtain sample audio and corresponding text and audio annotations, wherein the audio annotations are used to represent the audio features of the sample audio in a discretized form; perform a first training process on a language processing model based on the sample audio and the audio annotations to obtain a first audio processing model, wherein the first training process is used to enable the first audio processing model to process audio coding unit sequences; perform a second training process on the first audio processing model based on the sample audio and the text annotations to obtain a target audio processing model, wherein the second training process is used to enable the target audio processing model to process audio.

2. The method according to claim 1, characterized in that, The audio annotation includes the audio coding unit sequence of the sample audio; the audio coding unit sequence is obtained by encoding the sample audio and quantizing it in a preset number of dimensions; the language processing model includes an embedding layer and an audio output head; the number of embedding vectors in the embedding layer is set to the preset number, and the embedding layer is used to encode and generate embedding vector representations that characterize semantic features; The audio output head is used to perform audio coding unit sequence prediction processing on the semantic feature vector; The step of performing a first training process on the language processing model based on the sample audio and the audio annotation to obtain a first audio processing model includes: performing the first training process on the embedding layer and the audio output head of the language processing model based on the sample audio and the audio coding unit sequence to obtain the first audio processing model.

3. The method according to claim 2, characterized in that, The method further includes: replacing the text output header of the initial language processing model with the audio output header to obtain the language processing model; wherein the text output header is used to perform text encoding unit sequence prediction processing on the semantic feature vector.

4. The method according to claim 2, characterized in that, The sample audio includes a first audio segment and a second audio segment, the first audio segment corresponding to a first subsequence of the audio coding unit sequence, and the second audio segment corresponding to a second subsequence of the audio coding unit sequence; wherein, the first training process of the embedding layer and the audio output head of the language processing model based on the sample audio and the audio coding unit sequence to obtain the first audio processing model includes: inputting the first subsequence into the language processing model to obtain a first prediction sequence corresponding to the second audio segment; obtaining a first loss value between the second subsequence and the first prediction sequence; and adjusting the parameters of the embedding layer and the audio output head based on the first loss value to obtain the first audio processing model.

5. The method according to claim 1, characterized in that, The step of performing a second training process on the first audio processing model based on the sample audio and the text annotation to obtain a target audio processing model includes: inputting the sample audio into the second audio processing model to obtain the predicted recognition text of the sample audio; the second audio processing model includes a text output head, which is used to perform text encoding unit sequence prediction processing on the semantic feature vector; the second audio processing model is obtained based on the first audio processing model; obtaining a second loss value between the text annotation and the predicted recognition text; and adjusting the parameters of the text output head of the first audio processing model based on the second loss value to obtain the target audio processing model.

6. The method according to claim 5, characterized in that, The first audio processing model includes an audio output head, which is used to perform audio coding unit sequence prediction processing on semantic feature vectors; the method further includes: replacing the audio output head of the first audio processing model with the text output head to obtain the second audio processing model.

7. The method according to claim 1, characterized in that, The step of performing a second training process on the first audio processing model based on the sample audio and the text annotation to obtain a target audio processing model includes: performing a third training process on the first audio processing model based on the noisy sample audio corresponding to the sample audio and the audio annotation to obtain a third audio processing model; wherein, the third training process is used to enable the third audio processing model to have the ability to remove audio noise; the noisy sample audio is obtained by adding audio noise to the sample audio; and performing a fourth training process on the third audio processing model based on the sample audio and the text annotation to obtain the target audio processing model.

8. The method according to claim 7, characterized in that, The audio annotation includes the audio coding unit sequence of the sample audio; the noisy sample audio includes a third audio segment and a fourth audio segment, the fourth audio segment corresponding to a third sub-sequence of the audio coding unit sequence; the third training process of the first audio processing model based on the noisy sample audio corresponding to the sample audio and the audio annotation to obtain a third audio processing model includes: obtaining the noisy audio coding unit sequence of the third audio segment; inputting the noisy audio coding unit sequence into the first audio processing model to obtain a second prediction sequence corresponding to the fourth audio segment; obtaining a third loss value between the third sub-sequence and the second prediction sequence; adjusting the parameters of the first audio processing model based on the third loss value to obtain the third audio processing model.

9. A speech processing method, characterized in that, include: Acquire the speech to be processed; input the speech to be processed into a target audio processing model to obtain the speech processing result of the speech to be processed; wherein the target audio processing model is obtained based on the method described in any one of claims 1-8.

10. The method according to claim 9, characterized in that, The speech processing result includes at least one of the following: the recognized text corresponding to the speech to be processed, the response text corresponding to the speech to be processed, the response speech corresponding to the speech to be processed, the translated text in a first preset language corresponding to the speech to be processed, and the translated speech in a second preset language corresponding to the speech to be processed.

11. A model training device, characterized in that, include: An acquisition module is used to acquire sample audio and corresponding text and audio annotations, wherein the audio annotations are used to represent the audio features of the sample audio in a discretized form; a first training module is used to perform a first training process on a language processing model based on the sample audio and the audio annotations to obtain a first audio processing model, wherein the first training process is used to enable the first audio processing model to process audio coding unit sequences; a second training module is used to perform a second training process on the first audio processing model based on the sample audio and the text annotations to obtain a target audio processing model, wherein the second training process is used to enable the target audio processing model to process audio.

12. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method as described in any one of claims 1-8 or the speech processing method as described in claim 9 or 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method as described in any one of claims 1-8 or the speech processing method as described in claim 9 or 10.

14. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the model training method as described in any one of claims 1-8 or the speech processing method as described in claim 9 or 10.

Images