Voice large model modal alignment method and device based on two-stage decoupling mode

Abstract

The invention provides a voice large model modal alignment method and device based on a two-stage decoupling mode, and relates to the technical field of natural language processing. The method comprises the following steps: acquiring a pre-trained voice data set and a pre-trained task instruction text; constructing an initial large voice model, and pre-training the initial large voice model by adopting a two-stage decoupling mode according to the pre-trained voice data set and the pre-trained task instruction text to obtain a pre-trained large voice model; performing instruction fine tuning on the pre-trained voice large model by adopting a LoRA fine tuning technology to obtain a trained voice large model; and inputting to-be-processed voice data and an instruction corresponding to the voice data into the trained voice large model for processing, and outputting a text matched with an instruction demand corresponding to the voice data. By adopting the method, the problem of information loss caused by feature decoupling can be solved, and the task analysis performance of the large voice model can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and device for aligning large speech model modalities based on a two-stage decoupling approach. Background Art

[0002] With the rapid development of computer technology and artificial intelligence technology, speech processing technology has gradually transitioned from traditional signal processing methods to methods based on machine learning and deep learning algorithms as the mainstream stage. Traditional speech processing methods include: methods based on sound signal processing, methods based on feature extraction, and methods based on deep learning models. In the multimodal field, modal alignment is a highly concerned issue. In the multimodal field, modal alignment methods include: methods based on statistical learning, explicit alignment methods based on deep learning, implicit alignment methods based on deep learning, and other alignment methods based on deep learning. The purpose of modal alignment of large speech models is to enable the large language model trained based on a large text corpus at the back end to fully understand the content of the input speech and complete various speech-related downstream tasks in combination with the speech content. According to the differences in the selection of target downstream tasks, the specific technologies are mainly divided into the following two categories: alignment when the output modality contains only text and alignment when the output modality contains text and speech.

[0003] At present, there are two main problems that need to be solved in modal alignment on large speech models: (1) The length of speech representation is much longer than the length of text representation sequence. The longer sequence length will not only affect the accuracy of large model reasoning, but also increase the reasoning delay of the model. Therefore, the length of the input speech representation needs to be reduced; (2) There is a big difference between speech representation and text representation in the representation space. Speech representation is usually mapped to the input space of the large model through a trainable linear mapping module, but it requires high-quality and sufficiently large training data sets as support. The current method for modal alignment in large speech models is the SALMONN model jointly proposed by Tsinghua University and ByteDance. It uses two encoders for semantic features and acoustic features to encode speech, introduces speech-text modal alignment through the Q-Former module for image-text alignment, and extracts the required features from the speech features by using a preset trainable multiple Token sequence, and the length is consistent with the preset sequence length. Finally, the above method has achieved good performance on multiple downstream tasks.

[0004] Problems with existing large speech models include: insufficient decoupling. Speech information can usually be divided into semantic information and acoustic information. Different downstream tasks pay different attention to these two aspects of information. The existing technology only uses a single encoder and only focuses on one of the two aspects of information. The other part, like the SALMONN model, uses two encoders to encode the two aspects of features respectively, but they are coupled in an unclear way later, resulting in a mixture of information. Summary of the invention

[0005] In order to solve the technical problem that the prior art often only considers a single aspect of information or there is feature coupling that leads to information loss and thus fails to achieve sufficient speech information decoupling and re-extraction, the embodiment of the present invention provides a speech large model modal alignment method and device based on a two-stage decoupling method. The technical solution is as follows:

[0006] On the one hand, a speech large model modal alignment method based on a two-stage decoupling method is provided, the method is implemented by a speech large model modal alignment device based on a two-stage decoupling method, and the method includes:

[0007] S1, obtaining a pre-trained speech data set and a pre-trained task instruction text;

[0008] S2, using the Librosa method to clean the speech data set to obtain a cleaned speech data set;

[0009] S3, constructing an initial large speech model; the initial large speech model includes: a large language model and an alignment module;

[0010] S4, inputting the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, decoupling the cleaned speech data set through the alignment module to obtain the final speech features; using the word segmenter of the large language model to process the pre-trained task instruction text to obtain the pre-trained text features; processing the final speech features and the pre-trained text features through sequence-level splicing to obtain a final feature sequence; pre-training the initial speech large model according to the final feature sequence to obtain a pre-trained speech large model;

[0011] S5. Obtain a command fine-tuning training set; according to the command fine-tuning training set, use LoRA fine-tuning technology to perform command fine-tuning on the pre-trained voice model to obtain a trained voice model;

[0012] S6. Obtain the speech data to be processed and the instructions corresponding to the speech data; input the speech data to be processed and the instructions corresponding to the speech data into the trained speech model for processing, and output text that matches the instruction requirements corresponding to the speech data.

[0013] Optionally, the initial large speech model further includes: a speech encoder and an audio encoder;

[0014] The speech encoder is used to process the input data to obtain semantic features;

[0015] The audio encoder is used to process input data to obtain audio features.

[0016] Optionally, the alignment module comprises: a continuous integration excitation module, a convolutional projection layer, a desired content encoder, an undesired content encoder and a disentanglement module;

[0017] The continuous integration excitation module is used to reduce the length of the semantic features to obtain the reduced semantic features;

[0018] The convolutional projection layer is used to reduce the length of the audio features to obtain the reduced audio features;

[0019] Wherein, the required content encoder is used to encode language content information from speech input;

[0020] Wherein, the undesired content encoder is used to model non-linguistic language features;

[0021] Wherein, the de-entanglement module is used to separate different modal information.

[0022] Optionally, the step S4 inputs the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, and decouples the cleaned speech data set through the alignment module to obtain the final speech features, including:

[0023] S41, inputting the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, processing the cleaned speech data set through a speech encoder to obtain semantic features; processing the cleaned speech data set through an audio encoder to obtain audio features;

[0024] S42, inputting the semantic features into the continuous integration excitation module for length reduction processing to obtain reduced semantic features; inputting the audio features into the convolutional projection layer for length reduction processing to obtain reduced audio features;

[0025] S43, concatenating the reduced semantic features and the reduced audio features at a sequence level to obtain concatenated features; inputting the concatenated features into a required content encoder for processing to obtain a first concatenated feature; inputting the concatenated features into a non-required content encoder for processing to obtain a second concatenated feature;

[0026] S44: Input the first concatenated feature and the second concatenated feature into a de-entanglement module for decoupling processing to obtain the final speech feature.

[0027] Optionally, the step S4 pre-trains the initial speech model according to the final feature sequence to obtain a pre-trained speech model, including:

[0028] The final feature sequence is input into the large language model for decoding, and the decoding result is output; according to the decoding result and the cross entropy loss function, the initial large speech model is pre-trained to obtain a pre-trained large speech model.

[0029] Optionally, the instruction fine-tuning training set of S5 includes: a speech data set for instruction fine-tuning and a task instruction text corresponding to the speech data set.

[0030] Optionally, the step of S5 fine-tuning the pre-trained speech model according to the instruction fine-tuning training set by using the LoRA fine-tuning technology to obtain the trained speech model includes:

[0031] S51, inputting the speech data set of instruction fine-tuning and the task instruction text corresponding to the speech data set into the pre-trained speech large model, processing the task instruction text corresponding to the speech data set through the word segmenter of the large language model, and obtaining the text features of the instruction fine-tuning;

[0032] S52, processing the voice data set for instruction fine-tuning in a decoupling manner to obtain voice features for instruction fine-tuning;

[0033] S53. Concatenate the text features of instruction fine-tuning and the speech features of instruction fine-tuning at the sequence level to obtain a concatenated feature sequence; input the feature sequence into the large language model, perform instruction fine-tuning on the large language model through the LoRA fine-tuning technology, and obtain a trained speech large model.

[0034] On the other hand, a speech large model modal alignment device based on a two-stage decoupling method is provided, and the device is applied to a speech large model modal alignment method based on a two-stage decoupling method, and the device includes:

[0035] A first acquisition unit, used to acquire a pre-trained speech data set and a pre-trained task instruction text;

[0036] A second acquisition unit is used to clean the speech data set using the Librosa method to obtain a cleaned speech data set;

[0037] A construction unit, used to construct an initial large speech model; the initial large speech model includes: a large language model and an alignment module;

[0038] A pre-training unit is used to input the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, decouple the cleaned speech data set through the alignment module to obtain the final speech features; use the word segmenter of the large language model to process the pre-trained task instruction text to obtain the pre-trained text features; process the final speech features and the pre-trained text features through sequence-level splicing to obtain a final feature sequence; pre-train the initial speech large model according to the final feature sequence to obtain a pre-trained speech large model;

[0039] An instruction fine-tuning unit is used to obtain an instruction fine-tuning training set; according to the instruction fine-tuning training set, the pre-trained speech model is fine-tuned by using the LoRA fine-tuning technology to obtain a trained speech model;

[0040] The output unit is used to obtain the voice data to be processed and the instructions corresponding to the voice data; the voice data to be processed and the instructions corresponding to the voice data are input into the trained voice model for processing, and the text matching the instruction requirements corresponding to the voice data is output.

[0041] Optionally, the initial large speech model further includes: a speech encoder and an audio encoder;

[0042] The speech encoder is used to process the input data to obtain semantic features;

[0043] The audio encoder is used to process input data to obtain audio features.

[0044] Optionally, the alignment module comprises: a continuous integration excitation module, a convolutional projection layer, a desired content encoder, an undesired content encoder and a disentanglement module;

[0045] The continuous integration excitation module is used to reduce the length of the semantic features to obtain the reduced semantic features;

[0046] The convolutional projection layer is used to reduce the length of the audio features to obtain the reduced audio features;

[0047] Wherein, the required content encoder is used to encode language content information from speech input;

[0048] Wherein, the undesired content encoder is used to model non-linguistic language features;

[0049] Wherein, the de-entanglement module is used to separate different modal information.

[0050] Optionally, the cleaned speech data set and the pre-trained task instruction text are input into the initial speech large model, and the cleaned speech data set is decoupled by the alignment module to obtain the final speech features, including:

[0051] Inputting the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, processing the cleaned speech data set through a speech encoder to obtain semantic features; processing the cleaned speech data set through an audio encoder to obtain audio features;

[0052] The semantic features are input into the continuous integration excitation module for length reduction processing to obtain the reduced semantic features; the audio features are input into the convolutional projection layer for length reduction processing to obtain the reduced audio features;

[0053] The reduced semantic features and the reduced audio features are concatenated at the sequence level to obtain concatenated features; the concatenated features are input into the required content encoder for processing to obtain first concatenated features; the concatenated features are input into the non-required content encoder for processing to obtain second concatenated features;

[0054] The first concatenated feature and the second concatenated feature are input into the de-entanglement module for decoupling processing to obtain the final speech feature.

[0055] Optionally, pre-training the initial speech large model according to the final feature sequence to obtain the pre-trained speech large model includes:

[0056] The final feature sequence is input into the large language model for decoding, and the decoding result is output; according to the decoding result and the cross entropy loss function, the initial large speech model is pre-trained to obtain a pre-trained large speech model.

[0057] Optionally, the instruction fine-tuning training set includes: a speech data set for instruction fine-tuning and a task instruction text corresponding to the speech data set.

[0058] Optionally, the instruction fine-tuning unit is used to:

[0059] Inputting the speech data set of instruction fine-tuning and the task instruction text corresponding to the speech data set into the pre-trained speech large model, processing the task instruction text corresponding to the speech data set through the word segmenter of the large language model, and obtaining the text features of the instruction fine-tuning;

[0060] The speech data set of instruction fine-tuning is processed in a decoupled manner to obtain the speech features of instruction fine-tuning;

[0061] The text features of instruction fine-tuning and the speech features of instruction fine-tuning are concatenated at the sequence level to obtain a concatenated feature sequence; the feature sequence is input into the large language model, and the instruction fine-tuning of the large language model is performed through the LoRA fine-tuning technology to obtain a trained speech large model.

[0062] On the other hand, a speech large model modal alignment device based on a two-stage decoupling method is provided, and the speech large model modal alignment device based on a two-stage decoupling method comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned speech large model modal alignment methods based on a two-stage decoupling method is implemented.

[0063] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned speech large model modal alignment methods based on the two-stage decoupling method.

[0064] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0065] The embodiment of the present invention first obtains a pre-trained speech data set and a pre-trained task instruction text; uses the Librosa method to clean the speech data set to obtain a cleaned speech data set; secondly, constructs an initial speech large model; the initial speech large model includes: a large language model and an alignment module; the cleaned speech data set and the pre-trained task instruction text are input into the initial speech large model, and the cleaned speech data set is decoupled by the alignment module to obtain the final speech features; the pre-trained task instruction text is processed by the word segmenter of the large language model to obtain the pre-trained text features; The final speech features and pre-trained text features are processed in a splicing manner to obtain a final feature sequence; based on the final feature sequence, the initial speech large model is pre-trained to obtain a pre-trained speech large model; an instruction fine-tuning training set is obtained; based on the instruction fine-tuning training set, the pre-trained speech large model is fine-tuned using the LoRA fine-tuning technology to obtain a trained speech large model; finally, the speech data to be processed and the instructions corresponding to the speech data are obtained; the speech data to be processed and the instructions corresponding to the speech data are input into the trained speech large model for processing, and text matching the instruction requirements corresponding to the speech data is output.

[0066] In the modal alignment, the embodiment of the present invention uses two different encoders and different length reduction methods to perform the first stage decoupling, fully mines the required information from the input speech and decouples it to perform different degrees of length reduction. In the information integration process, the sequence-level splicing and disentanglement modules are used to perform the second stage decoupling, which reduces the original information loss in the information integration process and performs screening and filtering, and finally obtains the input representation that meets the downstream task requirements. The present invention can solve the problem of information loss caused by feature decoupling and improve the performance of comprehensive speech understanding and analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 It is a structural schematic diagram of a speech large model modal alignment method based on a two-stage decoupling method provided by an embodiment of the present invention;

[0069] Figure 2 It is a flow chart of a method for aligning speech large models based on a two-stage decoupling method provided by an embodiment of the present invention;

[0070] Figure 3 It is a block diagram of a speech large model modal alignment device based on a two-stage decoupling method provided by an embodiment of the present invention;

[0071] Figure 4 It is a structural schematic diagram of a speech large model modal alignment device based on a two-stage decoupling method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0073] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0074] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0075] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0076] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0077] The embodiment of the present invention provides a method for aligning the modalities of a large speech model based on a two-stage decoupling method. The method can be implemented by a device for aligning the modalities of a large speech model based on a two-stage decoupling method. The device for aligning the modalities of a large speech model based on a two-stage decoupling method can be a terminal or a server. Figure 1It is a structural schematic diagram of a speech large model modal alignment method based on a two-stage decoupling method provided by an embodiment of the present invention; in a feasible implementation manner, the acquired speech data and the task instruction text corresponding to the speech data are input into the initially constructed speech large model, and the speech data is processed by a speech encoder to obtain semantic features; the speech data is processed by an audio encoder to obtain audio features; the semantic features are input into a continuous integration excitation module for length reduction processing to obtain reduced semantic features; the audio features are input into a convolutional projection layer for length reduction processing to obtain reduced audio features; the reduced semantic features and the reduced audio features are sequence-level spliced ​​to obtain the spliced ​​result; the spliced ​​result is input into a The first encoding result is input into the required content encoder for encoding processing to obtain a first encoding result; the concatenated result is input into the non-required content encoder for encoding processing to obtain a second encoding result; the first encoding result and the second encoding result are input into the disentanglement module to output the final speech features; the word segmenter of the back-end large model is used to process the input task instruction text to obtain the task instruction text features; wherein the back-end large model is a large language model; according to the final speech features and the task instruction text features, the back-end large model is input for training, and the initial speech large model is further trained to obtain a pre-trained speech large model; according to the pre-trained speech large model, the LoRA fine-tuning technology is used to perform instruction fine-tuning training on the pre-trained speech large model to obtain a trained speech large model.

[0078] like Figure 2 The flowchart of the speech large model modal alignment method based on the two-stage decoupling method is shown in the figure. The processing flow of the method may include the following steps:

[0079] S1. Obtain a pre-trained speech dataset and pre-trained task instruction text.

[0080] In a feasible implementation, the pre-trained speech data set may be a speech downstream task data set including: a data set in speech processing related application scenarios such as speech recognition, speech translation, and speech emotion recognition.

[0081] S2. Use the Librosa method to clean the speech data set to obtain a cleaned speech data set.

[0082] In a feasible implementation, the present application may use the Librosa method, the Soundfile method, and the Montreal Forced Aligner method to clean the pre-trained speech data set.

[0083] S3. Build an initial large speech model; the initial large speech model includes: a large language model and an alignment module.

[0084] Among them, a large language model refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text.

[0085] Optionally, the initial large speech model further includes: a speech encoder and an audio encoder;

[0086] Among them, the speech encoder is used to process the input data to obtain semantic features;

[0087] Among them, the audio encoder is used to process the input data to obtain audio features.

[0088] Among them, the alignment module is used to establish a corresponding relationship between different modal data.

[0089] In a feasible implementation, by encoding the input speech using a speech encoder and an audio encoder respectively, the semantic information and acoustic information of the speech can be considered simultaneously, and the decoupling of the encoding end can be effectively achieved, which helps the model to extract more effective information from the speech.

[0090] Optionally, the alignment module comprises: a continuous integration excitation module, a convolutional projection layer, a desired content encoder, an undesired content encoder, and a disentanglement module;

[0091] The continuous integration excitation module is used to reduce the length of the semantic features to obtain the reduced semantic features;

[0092] The convolutional projection layer is used to reduce the length of the audio features to obtain the reduced audio features.

[0093] Wherein, the required content encoder is used to encode language content information from the speech input;

[0094] Among them, the non-desired content encoder is used to model non-linguistic language features;

[0095] Among them, the disentanglement module is used to separate different modal information.

[0096] Among them, the disentanglement module can reduce the correlation between features and improve the performance and robustness of large speech models.

[0097] In a feasible implementation, two length reduction methods, namely, continuous integration excitation module and convolutional projection layer, are used to reduce the length of feature sequences for different coding information, thereby effectively shortening the length of the sequence finally input into the large language model and improving the accuracy and efficiency of model reasoning.

[0098] In a feasible implementation, different feature representation sequence dimensions are spliced ​​during information integration, reducing the loss of the extracted original information, and a de-entanglement module is used to complete the second stage of decoupling. The integrated information is further screened and filtered for downstream tasks, retaining the information most suitable for the current downstream task, thereby improving the performance of downstream tasks.

[0099] S4. Input the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, process the cleaned speech data set in a decoupled manner to obtain the final speech features; use the word segmenter of the large language model to process the pre-trained task instruction text to obtain the pre-trained text features; process the final speech features and pre-trained text features in a sequence-level concatenation manner to obtain the final feature sequence; pre-train the initial speech large model according to the final feature sequence to obtain the pre-trained speech large model.

[0100] Among them, the pre-training loss function is the cross entropy loss function.

[0101] In a feasible implementation, a cross entropy loss function and an AdamW optimizer are used to train and optimize model parameters.

[0102] Among them, the present application adopts a two-stage decoupling method when processing voice input, wherein the first stage of decoupling includes: using a voice encoder and an audio encoder to encode the input voice data to obtain semantic features and audio features, using a continuous integration excitation module and a convolutional projection layer to perform length compression processing on the semantic features and audio features respectively, and projecting them to the same dimension for sequence-level splicing to obtain a splicing result;

[0103] Among them, the two-stage decoupling includes: using an alignment module to process the splicing results to obtain the final speech features.

[0104] Optionally, S4 inputs the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, and decouples the cleaned speech data set through the alignment module to obtain the final speech features, including:

[0105] S41, inputting the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, processing the cleaned speech data set through a speech encoder to obtain semantic features; processing the cleaned speech data set through an audio encoder to obtain audio features;

[0106] S42, inputting the semantic features into the continuous integration excitation module for length reduction processing to obtain reduced semantic features; inputting the audio features into the convolutional projection layer for length reduction processing to obtain reduced audio features;

[0107] S43, concatenating the reduced semantic features and the reduced audio features at a sequence level to obtain concatenated features; inputting the concatenated features into a required content encoder for processing to obtain a first concatenated feature; inputting the concatenated features into a non-required content encoder for processing to obtain a second concatenated feature;

[0108] S44: Input the first concatenated feature and the second concatenated feature into a de-entanglement module for decoupling processing to obtain the final speech feature.

[0109] Optionally, S4 pre-trains the initial speech model according to the final feature sequence to obtain a pre-trained speech model, including:

[0110] The final feature sequence is input into the large language model for decoding, and the decoding result is output; based on the decoding result and the cross entropy loss function, the initial large speech model is pre-trained to obtain a pre-trained large speech model.

[0111] Among them, during the pre-training process, only the alignment module is trained and the large speech model is frozen, in order to provide good initial parameters for the alignment module.

[0112] S5. Obtain a command fine-tuning training set. According to the command fine-tuning training set, use LoRA fine-tuning technology to perform command fine-tuning on the pre-trained speech model to obtain a trained speech model.

[0113] Optionally, the instruction fine-tuning training set of S5 includes: a speech data set for instruction fine-tuning and a task instruction text corresponding to the speech data set.

[0114] Optionally, the specific implementation process of S5 includes S51-S53:

[0115] S51, inputting the speech data set of instruction fine-tuning and the task instruction text corresponding to the speech data set into the pre-trained speech model, processing the task instruction text corresponding to the speech data set through the word segmenter of the large language model, and obtaining the text features of the instruction fine-tuning;

[0116] S52, processing the voice data set for instruction fine-tuning in a decoupling manner to obtain voice features for instruction fine-tuning;

[0117] S53. Concatenate the text features of the instruction fine-tuning and the speech features of the instruction fine-tuning at the sequence level to obtain a concatenated feature sequence; input the feature sequence into the large language model, perform instruction fine-tuning on the large language model through the LoRA fine-tuning technology, and obtain a trained speech large model.

[0118] In a feasible implementation, a speech training set for instruction fine-tuning training and a task instruction text corresponding to the speech training set are obtained; the task instruction text and the speech data are subjected to the same speech encoding and two-stage decoupling process as in the pre-training process, and then input into a large language model, and a result that meets the instruction requirements is output, and the model parameters are optimized using the same cross entropy loss and optimizer as in the pre-training, wherein the large language model is no longer frozen during the instruction fine-tuning training process, and the model parameters are fine-tuned using the LoRA fine-tuning technology of the Peft library to obtain a trained speech large model.

[0119] S6. Obtain the speech data to be processed and the instructions corresponding to the speech data; input the speech data to be processed and the instructions corresponding to the speech data into the trained speech model for processing, and output text that matches the instruction requirements corresponding to the speech data.

[0120] In a feasible implementation, the voice data to be processed and the task instruction text corresponding to the voice data are obtained; the voice data to be processed and the task instruction text corresponding to the voice data are input into a trained voice model for processing, and a text matching the instruction requirements is output. For example, a voice and the text of the voice translation instruction are input into a trained voice model, and the text result after the actual content of the voice is translated into the target language is output.

[0121] The embodiment of the present invention first obtains a pre-trained speech data set and a pre-trained task instruction text; uses the Librosa method to clean the speech data set to obtain a cleaned speech data set; secondly, constructs an initial speech large model; the initial speech large model includes: a large language model and an alignment module; the cleaned speech data set and the pre-trained task instruction text are input into the initial speech large model, and the cleaned speech data set is decoupled by the alignment module to obtain the final speech features; the pre-trained task instruction text is processed by the word segmenter of the large language model to obtain the pre-trained text features; The final speech features and pre-trained text features are processed in a splicing manner to obtain a final feature sequence; based on the final feature sequence, the initial speech large model is pre-trained to obtain a pre-trained speech large model; an instruction fine-tuning training set is obtained; based on the instruction fine-tuning training set, the pre-trained speech large model is fine-tuned using the LoRA fine-tuning technology to obtain a trained speech large model; finally, the speech data to be processed and the instructions corresponding to the speech data are obtained; the speech data to be processed and the instructions corresponding to the speech data are input into the trained speech large model for processing, and text matching the instruction requirements corresponding to the speech data is output.

[0122] In the modal alignment, the embodiment of the present invention uses two different encoders and different length reduction methods to perform the first stage decoupling, fully mines the required information from the input speech and decouples it to perform different degrees of length reduction. In the information integration process, the sequence-level splicing and disentanglement modules are used to perform the second stage decoupling, which reduces the original information loss in the information integration process and performs screening and filtering, and finally obtains the input representation that meets the downstream task requirements. The present invention can solve the problem of information loss caused by feature decoupling and improve the performance of comprehensive speech understanding and analysis tasks.

[0123] Figure 3 The present invention is a block diagram of a speech large model modal alignment device based on a two-stage decoupling method according to an exemplary embodiment. The device is used in a speech large model modal alignment method based on a two-stage decoupling method. Figure 3 The device includes a first acquisition unit 310, a second acquisition unit 320, a construction unit 330, a pre-training unit 340, an instruction fine-tuning unit 350 and an output unit 360. Wherein:

[0124] A first acquisition unit 310 is used to acquire a pre-trained speech data set and a pre-trained task instruction text;

[0125] A second acquisition unit 320, configured to clean the speech data set using the Librosa method to obtain a cleaned speech data set;

[0126] A construction unit 330 is used to construct an initial large speech model; the initial large speech model includes: a large language model and an alignment module;

[0127] The pre-training unit 340 is used to input the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, decouple the cleaned speech data set through the alignment module to obtain the final speech features; use the word segmenter of the large language model to process the pre-trained task instruction text to obtain the pre-trained text features; process the final speech features and the pre-trained text features through sequence-level splicing to obtain the final feature sequence; pre-train the initial speech large model according to the final feature sequence to obtain the pre-trained speech large model;

[0128] The instruction fine-tuning unit 350 is used to obtain an instruction fine-tuning training set; according to the instruction fine-tuning training set, the LoRA fine-tuning technology is used to perform instruction fine-tuning on the pre-trained speech large model to obtain a trained speech large model;

[0129] The output unit 360 is used to obtain the voice data to be processed and the instructions corresponding to the voice data; input the voice data to be processed and the instructions corresponding to the voice data into the trained voice model for processing, and output text that matches the instruction requirements corresponding to the voice data.

[0130] Optionally, the initial large speech model further includes: a speech encoder and an audio encoder;

[0131] The speech encoder is used to process the input data to obtain semantic features;

[0132] The audio encoder is used to process input data to obtain audio features.

[0133] Optionally, the alignment module includes: a continuous integration excitation module, a convolutional projection layer, a desired content encoder, an undesired content encoder, and a disentanglement module;

[0134] The continuous integration excitation module is used to reduce the length of the semantic features to obtain the reduced semantic features;

[0135] The convolutional projection layer is used to reduce the length of the audio features to obtain the reduced audio features;

[0136] Wherein, the required content encoder is used to encode language content information from speech input;

[0137] Wherein, the undesired content encoder is used to model non-linguistic language features;

[0138] Wherein, the de-entanglement module is used to separate different modal information.

[0139] Optionally, the cleaned speech data set and the pre-trained task instruction text are input into the initial speech large model, and the cleaned speech data set is decoupled by the alignment module to obtain the final speech features, including:

[0140] Inputting the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, processing the cleaned speech data set through a speech encoder to obtain semantic features; processing the cleaned speech data set through an audio encoder to obtain audio features;

[0141] The semantic features are input into the continuous integration excitation module for length reduction processing to obtain the reduced semantic features; the audio features are input into the convolutional projection layer for length reduction processing to obtain the reduced audio features;

[0142] The reduced semantic features and the reduced audio features are concatenated at the sequence level to obtain concatenated features; the concatenated features are input into the required content encoder for processing to obtain first concatenated features; the concatenated features are input into the non-required content encoder for processing to obtain second concatenated features;

[0143] The first concatenated feature and the second concatenated feature are input into the de-entanglement module for decoupling processing to obtain the final speech feature.

[0144] Optionally, pre-training the initial speech large model according to the final feature sequence to obtain the pre-trained speech large model includes:

[0145] The final feature sequence is input into the large language model for decoding, and the decoding result is output; according to the decoding result and the cross entropy loss function, the initial large speech model is pre-trained to obtain a pre-trained large speech model.

[0146] Optionally, the instruction fine-tuning training set includes: a speech data set for instruction fine-tuning and a task instruction text corresponding to the speech data set.

[0147] Optionally, the instruction fine-tuning unit 350 is used to:

[0148] Inputting the speech data set of instruction fine-tuning and the task instruction text corresponding to the speech data set into the pre-trained speech large model, processing the task instruction text corresponding to the speech data set through the word segmenter of the large language model, and obtaining the text features of the instruction fine-tuning;

[0149] The speech data set of instruction fine-tuning is processed in a decoupled manner to obtain the speech features of instruction fine-tuning;

[0150] The text features of instruction fine-tuning and the speech features of instruction fine-tuning are concatenated at the sequence level to obtain a concatenated feature sequence; the feature sequence is input into the large language model, and the instruction fine-tuning of the large language model is performed through the LoRA fine-tuning technology to obtain a trained speech large model.

[0151] The embodiment of the present invention first obtains a pre-trained speech data set and a pre-trained task instruction text; uses the Librosa method to clean the speech data set to obtain a cleaned speech data set; secondly, constructs an initial speech large model; the initial speech large model includes: a large language model and an alignment module; the cleaned speech data set and the pre-trained task instruction text are input into the initial speech large model, and the cleaned speech data set is decoupled by the alignment module to obtain the final speech features; the pre-trained task instruction text is processed by the word segmenter of the large language model to obtain the pre-trained text features; The final speech features and pre-trained text features are processed in a splicing manner to obtain a final feature sequence; based on the final feature sequence, the initial speech large model is pre-trained to obtain a pre-trained speech large model; an instruction fine-tuning training set is obtained; based on the instruction fine-tuning training set, the pre-trained speech large model is fine-tuned using the LoRA fine-tuning technology to obtain a trained speech large model; finally, the speech data to be processed and the instructions corresponding to the speech data are obtained; the speech data to be processed and the instructions corresponding to the speech data are input into the trained speech large model for processing, and text matching the instruction requirements corresponding to the speech data is output.

[0152] In the modal alignment, the embodiment of the present invention uses two different encoders and different length reduction methods to perform the first stage decoupling, fully mines the required information from the input speech and decouples it to perform different degrees of length reduction. In the information integration process, the sequence-level splicing and disentanglement modules are used to perform the second stage decoupling, which reduces the original information loss in the information integration process and performs screening and filtering, and finally obtains the input representation that meets the downstream task requirements. The present invention can solve the problem of information loss caused by feature decoupling and improve the performance of comprehensive speech understanding and analysis tasks.

[0153] Figure 4is a structural diagram of a speech large model modal alignment device based on a two-stage decoupling method provided by an embodiment of the present invention, such as Figure 4 As shown, the speech large model modal alignment device based on the two-stage decoupling method may include the above Figure 3 The device for aligning the modalities of a speech large model based on a two-stage decoupling method is shown. Optionally, the device for aligning the modalities of a speech large model based on a two-stage decoupling method 410 may include a first processor 2001 .

[0154] Optionally, the speech large model modal alignment device 410 based on the two-stage decoupling method may also include a memory 2002 and a transceiver 2003 .

[0155] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0156] Combine the following Figure 4 The components of the speech large model modal alignment device 410 based on the two-stage decoupling method are specifically introduced:

[0157] The first processor 2001 is the control center of the speech large model modal alignment device 410 based on the two-stage decoupling method, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0158] Optionally, the first processor 2001 can perform various functions of the speech large model modal alignment device 410 based on the two-stage decoupling method by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0159] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0160] In a specific implementation, as an embodiment, the speech large model modality alignment device 410 based on the two-stage decoupling method may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0161] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0162] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0163] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0164] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0165] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0166] It should be noted that Figure 4 The structure of the speech large model modal alignment device 410 based on the two-stage decoupling method shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0167] In addition, the technical effects of the speech large model modal alignment device 410 based on the two-stage decoupling method can refer to the technical effects of the speech large model modal alignment method based on the two-stage decoupling method described in the above method embodiment, and will not be repeated here.

[0168] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0169] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0170] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0171] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0172] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0173] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0174] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0175] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0176] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0179] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0180] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A speech large model modal alignment method based on a two-stage decoupling method, characterized in that: The method comprises: S1, obtaining a pre-trained speech data set and a pre-trained task instruction text; S2, using the Librosa method to clean the speech data set to obtain a cleaned speech data set; S3, constructing an initial large speech model; the initial large speech model includes: a large language model and an alignment module; S4, inputting the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, decoupling the cleaned speech data set through the alignment module to obtain the final speech features; using the word segmenter of the large language model to process the pre-trained task instruction text to obtain the pre-trained text features; processing the final speech features and the pre-trained text features through sequence-level splicing to obtain a final feature sequence; pre-training the initial speech large model according to the final feature sequence to obtain a pre-trained speech large model; S5. Obtain a command fine-tuning training set; according to the command fine-tuning training set, use LoRA fine-tuning technology to perform command fine-tuning on the pre-trained voice model to obtain a trained voice model; S6. Obtain the speech data to be processed and the instructions corresponding to the speech data; input the speech data to be processed and the instructions corresponding to the speech data into the trained speech model for processing, and output text that matches the instruction requirements corresponding to the speech data.

2. The speech large model modal alignment method based on the two-stage decoupling method according to claim 1 is characterized in that: The initial speech model further includes: a speech encoder and an audio encoder; The speech encoder is used to process the input data to obtain semantic features; The audio encoder is used to process input data to obtain audio features.

3. The speech large model modal alignment method based on the two-stage decoupling method according to claim 1 is characterized in that: The alignment module includes: a continuous integration excitation module, a convolutional projection layer, a desired content encoder, an undesired content encoder, and a disentanglement module; The continuous integration excitation module is used to reduce the length of the semantic features to obtain the reduced semantic features; The convolutional projection layer is used to reduce the length of the audio features to obtain the reduced audio features; Wherein, the required content encoder is used to encode language content information from speech input; Wherein, the undesired content encoder is used to model non-linguistic language features; Wherein, the de-entanglement module is used to separate different modal information.

4. The speech large model modal alignment method based on the two-stage decoupling method according to claim 3 is characterized in that: The step S4 inputs the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, and decouples the cleaned speech data set through the alignment module to obtain the final speech features, including: S41, inputting the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, processing the cleaned speech data set through a speech encoder to obtain semantic features; processing the cleaned speech data set through an audio encoder to obtain audio features; S42, inputting the semantic features into the continuous integration excitation module for length reduction processing to obtain reduced semantic features; inputting the audio features into the convolutional projection layer for length reduction processing to obtain reduced audio features; S43, concatenating the reduced semantic features and the reduced audio features at a sequence level to obtain concatenated features; inputting the concatenated features into a required content encoder for processing to obtain a first concatenated feature; inputting the concatenated features into a non-required content encoder for processing to obtain a second concatenated feature; S44: Input the first concatenated feature and the second concatenated feature into a de-entanglement module for decoupling processing to obtain the final speech feature.

5. The speech large model modal alignment method based on the two-stage decoupling method according to claim 1 is characterized in that: The step S4 pre-trains the initial speech model according to the final feature sequence to obtain a pre-trained speech model, including: The final feature sequence is input into the large language model for decoding, and the decoding result is output; according to the decoding result and the cross entropy loss function, the initial large speech model is pre-trained to obtain a pre-trained large speech model.

6. The speech large model modal alignment method based on the two-stage decoupling method according to claim 1 is characterized in that: The instruction fine-tuning training set of S5 includes: a speech data set for instruction fine-tuning and a task instruction text corresponding to the speech data set.

7. The speech large model modal alignment method based on the two-stage decoupling method according to claim 6 is characterized in that: The step S5 fine-tunes the pre-trained speech model according to the instruction fine-tuning training set by using the LoRA fine-tuning technology to obtain the trained speech model, including: S51, inputting the speech data set of instruction fine-tuning and the task instruction text corresponding to the speech data set into the pre-trained speech large model, processing the task instruction text corresponding to the speech data set through the word segmenter of the large language model, and obtaining the text features of the instruction fine-tuning; S52, processing the voice data set for instruction fine-tuning in a decoupling manner to obtain voice features for instruction fine-tuning; S53. Concatenate the text features of instruction fine-tuning and the speech features of instruction fine-tuning at the sequence level to obtain a concatenated feature sequence; input the feature sequence into the large language model, perform instruction fine-tuning on the large language model through the LoRA fine-tuning technology, and obtain a trained speech large model.

8. A speech large model modal alignment device based on a two-stage decoupling method, wherein the speech large model modal alignment device based on a two-stage decoupling method is used to implement the speech large model modal alignment method based on a two-stage decoupling method as claimed in any one of claims 1 to 7, characterized in that: The device comprises: A first acquisition unit, used to acquire a pre-trained speech data set and a pre-trained task instruction text; A second acquisition unit is used to clean the speech data set using the Librosa method to obtain a cleaned speech data set; A construction unit, used to construct an initial large speech model; the initial large speech model includes: a large language model and an alignment module; A pre-training unit is used to input the cleaned speech data set and the pre-trained task instruction text into the initial speech large model, decouple the cleaned speech data set through the alignment module to obtain the final speech features; use the word segmenter of the large language model to process the pre-trained task instruction text to obtain the pre-trained text features; process the final speech features and the pre-trained text features through sequence-level splicing to obtain a final feature sequence; pre-train the initial speech large model according to the final feature sequence to obtain a pre-trained speech large model; An instruction fine-tuning unit is used to obtain an instruction fine-tuning training set; according to the instruction fine-tuning training set, the pre-trained speech model is fine-tuned by using the LoRA fine-tuning technology to obtain a trained speech model; The output unit is used to obtain the voice data to be processed and the instructions corresponding to the voice data; the voice data to be processed and the instructions corresponding to the voice data are input into the trained voice model for processing, and the text matching the instruction requirements corresponding to the voice data is output.

9. A speech large model modal alignment device based on a two-stage decoupling method, characterized in that: The speech large model modal alignment device based on the two-stage decoupling method includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images