Modal distortion speech recognition method and system

Abstract

The invention discloses a modal distortion speech recognition method and system. The method comprises the following steps: calculating soft distribution of audio-visual characteristics through a cluster weight extraction network, and generating a prompt vector of each layer in combination with a clustering center matrix; inserting the prompt vector into the self-attention layer by adopting a prefix prompt strategy or a preposed prompt strategy; an output sequence is generated in a feature decoder through autoregression, and cross entropy loss between the output sequence and a real label is calculated; splitting the prompt vector of each task into a general subspace representation and a specific subspace representation, and performing contrast loss calculation on the specific subspace representations of the three tasks; and carrying out weighted fusion on the cross entropy loss and the comparison loss, then carrying out back propagation, and jointly updating parameters of all prompt vectors and language models. According to the method, the difference between prompts of different modal tasks is increased while the difference between modal missing prompts and undistorted prompts is reduced, so that the recognition effect of the audio-video multi-modal language model under different modal tasks is improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning technology, and in particular to a method and system for modal distortion speech recognition. Background Art

[0002] Audio and video speech recognition utilizes the synergy between speech signals and lip-synced videos to generate natural language text, demonstrating high accuracy in multimodal learning. However, in real-world scenarios, due to limitations in recording equipment and environments, data is often subject to modal distortion (such as audio and video asynchrony, audio noise, or video blur), severely degrading the performance of existing audio and video speech recognition models.

[0003] In recent years, prompt learning, as a lightweight fine-tuning method, has achieved remarkable success in the field of natural language processing. By designing input prompts for pre-trained models, they can be quickly adapted to specific task requirements while retaining the model's pre-trained knowledge. In the field of multimodal learning, the application of prompt learning is still in its infancy. Traditional prompt learning methods mainly focus on unimodal scenarios. However, designing efficient prompt mechanisms to cope with the complexity of multimodal data under conditions of modal distortion remains an urgent problem. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the above-mentioned prior art and provide a modal distortion speech recognition method and system. The method is based on a new framework of contrastive decomposition and clustering prompts, which enhances the robustness of the audio-video multimodal language model under multimodal distortion conditions through prompt learning, and constrains the prompt features of each modal distortion through contrastive loss. While reducing the difference between modal missing prompts and distortion-free prompts, the difference between prompts of different modal distortion tasks is increased, thereby significantly improving the recognition effect of the audio-video multimodal language model under different modal distortion tasks.

[0005] To this end, the present invention adopts the following technical solution: a modal distortion speech recognition method, which includes the steps of: 1) Load the pre-trained audio encoder, video encoder, audio and video feature fusion module, feature encoder, and feature decoder into the audio-video multimodal language model, and load the alignment parameters; 2) Using an audio encoder and a video encoder to simultaneously encode the audio and video inputs of the audio-video multimodal language model, and fusing them in an audio-video feature fusion module to obtain temporal features. The temporal features are then encoded by a feature encoder to obtain audio-visual features. 3) Calculate the soft distribution of audiovisual features through the cluster weight extraction network, and generate the prompt vector of each layer in combination with the cluster center matrix; 4) Using a prefix cue strategy or a pre-cue strategy, the cue vector is inserted into the self-attention layer before performing self-attention calculations; before inserting the cue vector into the self-attention layer, the cluster center matrix, the general subspace representation, and the specific subspace representations for the distortion-free task, the audio-only distortion task, and the video-only distortion task are initialized. 5) Autoregressively generate the output sequence in the feature decoder and calculate the cross entropy loss between it and the true label; 6) Split the cue vector of each task into a universal subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three tasks; 7) Weighted fusion of cross entropy loss and contrastive loss, followed by backpropagation to jointly update all cue vectors and the remaining parameters of the audio-video multimodal language model; 8) Repeat the above steps until the performance on the validation set converges or the predetermined number of training rounds is reached.

[0006] This paper uses clustering to process inputs from various modal tasks, generating different cue vectors that are inserted into the input or intermediate data of the self-attention layer, guiding the audio-video multimodal language model to adapt to different modal tasks. To further optimize the cue, the paper uses low-rank decomposition to decompose the cue vector into a universal subspace representation and a specific subspace representation, and uses contrastive loss to constrain the cue features of each modal task. This paper aims to improve the ability of the audio-video multimodal language model to perform speech recognition in various modal tasks.

[0007] Furthermore, step 3) specifically includes: 3.1) Cluster the audiovisual features in the time series dimension to obtain the membership distribution of the audiovisual features :

[0008] in, Represents clustering operation; Indicates audiovisual characteristics; represents the field of real numbers; T Indicates the length of the time series; D Represents the dimension of the feature; 3.2) Obtain the pair through linear mapping The score vector of the i-th cluster center among the cluster centers :

[0009] in, are learnable scoring weights, is a learnable rating bias; 3.3) Normalize the score vector Softmax to the soft membership weight of the cluster : in, represents the normalization function, Z k represents the score vector of the k-th cluster center, express Cluster centers j The degree of membership; 3.4) Using the soft membership weights The weighted summation of the prompt segments corresponding to the cluster centers generates a prompt vector that matches the current modality task: in, is the prompt vector of the i-th layer, is the cluster center matrix, , Indicates the j The prompt fragments corresponding to the cluster centers, L p Indicates the length of the prompt segment, Represents a composite function for scoring, Softmax, and calculating soft membership weights.

[0010] Furthermore, the prompt strategy in step 4) includes: 4.1) Pre-hint strategy: The generated hint vector is directly concatenated with the input sequence of the audio-video multimodal language model to form enhanced input data, which is inserted into the query, key, and value of the self-attention layer.

[0011] Furthermore, the prompt strategy in step 4) includes: 4.2) Prefix Hint Strategy: When generating the hint vector, instead of just generating the hint vector, we generate both the key hint and the value hint, which are then inserted into the key and value of the self-attention layer respectively.

[0012] Furthermore, in step 5), the calculation formula of the cross entropy loss is as follows: in, is the cross entropy loss, Before t- 1 language model input corresponding to the correct text, f Represents the output sequence generated by the feature decoder autoregression, the p in logp represents the conditional probability, and the summation symbol s Indicates the sequence length of the language model input.

[0013] Furthermore, in step 6), the prompt vector of each task is decomposed into a universal subspace representation and three task-specific subspace representations, and the decomposition formula is: , in, represents the universal subspace representation, represents the specific subspace representation of the current modality task n, Represents the prompt vector for the current modality task n.

[0014] Furthermore, the calculation formula of the contrast loss is: in, is the temperature coefficient, represents the similarity between vectors, represents the specific subspace representation of the current modality task n, Represents a specific subspace representation of positive sample tasks that are close to the current modality task, represents the task-specific subspace representation of the remaining negative samples, Represents a collection of modal task types, c represents the distortion-free task, ad represents the audio distortion task only, vd represents only the video distortion task; Represents a set of tasks that are similar to the current modal task n.

[0015] Furthermore, in step 7), the formula for weighted fusion cross entropy loss and contrast loss is:

[0016] in, is the cross entropy loss, is the contrast loss, is the loss weight coefficient.

[0017] Furthermore, all cue vectors and the remaining parameters of the audio-video multimodal language model are jointly updated via a stochastic gradient descent optimization algorithm.

[0018] The present invention also provides a modal distortion speech recognition system for implementing the above-mentioned modal distortion speech recognition method, comprising: Loading unit: loads the pre-trained audio encoder, video encoder, audio and video feature fusion module, feature encoder, and feature decoder into the audio-video multimodal language model, and loads the alignment parameters; Audio-visual feature acquisition unit: This unit uses an audio encoder and a video encoder to simultaneously encode the audio and video inputs of the audio-video multimodal language model. These are then fused in the audio-video feature fusion module to obtain temporal features, which are then encoded by the feature encoder to obtain audio-visual features. Hint vector generation unit: used to calculate the soft distribution of audiovisual features through the cluster weight extraction network, and generate the hint vector of each layer in combination with the cluster center matrix; Hint strategy insertion unit: Use prefix hint strategy or pre-hint strategy to insert hint vector into the self-attention layer, and then perform self-attention calculation; before inserting into the self-attention layer, initialize the cluster center matrix, general subspace representation, and specific subspace representations under the distortion-free task, audio distortion-only task, and video distortion-only task; Cross-entropy loss calculation unit: used to autoregressively generate the output sequence in the feature decoder and calculate the cross-entropy loss between it and the true label; Contrastive loss calculation unit: used to split the prompt vector of each task into a universal subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three tasks; Total loss calculation unit: used to weight the fusion of cross entropy loss and contrast loss, and then perform backpropagation to jointly update all prompt vectors and the remaining parameters of the audio-video multimodal language model; Loop unit: The above units are executed cyclically until the performance on the validation set converges or the predetermined number of training rounds is reached.

[0019] The beneficial effects of the present invention are as follows: the present invention enhances the robustness of the audio-video multimodal language model in multimodal tasks through prompt learning, constrains the prompt features of each modal task through contrast loss, reduces the difference between modality missing prompts and distortion-free prompts, and increases the difference between prompts of different modal tasks, thereby significantly improving the recognition effect of the audio-video multimodal language model in different modal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a modal distortion speech recognition method of the present invention; Figure 2 This is a composition diagram of a modal distortion speech recognition system of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific implementation methods.

[0022] Example 1 like Figure 1 As shown in FIG, the present invention is a method for modal distortion speech recognition based on contrast decomposition and clustering prompts, such as Figure 1 As shown, the steps are as follows: Step 1) Load the pre-trained audio encoder into the audio-video multimodal language model , video encoder , audio and video feature fusion module AVFusion, feature encoder and feature decoder EncoderLayer, and load their alignment parameters.

[0023] In step 2, the audio encoder and video encoder are used to simultaneously encode the audio and video input of the audio-video multimodal language model, and the temporal features are obtained by fusion in the audio and video feature fusion module. The temporal features are then encoded by the feature encoder to obtain audio-visual features.

[0024] Encode audio and video into features separately and fuse them:

[0025] in, represents the time series features after fusion, Represents audio-visual characteristics, represents the field of real numbers; T Indicates the length of the time series; D Represents the dimension of the feature, Indicates audio input; v Indicates video input.

[0026] Step 3) Calculate the soft distribution of audiovisual features through the cluster weight extraction network, and generate the prompt vector of each layer in combination with the cluster center matrix.

[0027] 3.1) Cluster the audiovisual features in the time series dimension to obtain the membership distribution of the audiovisual features :

[0028] in, Represents clustering operation; Indicates audiovisual characteristics; represents the field of real numbers; T Indicates the length of the time series; D Represents the dimension of the feature; 3.2) Obtain the pair through linear mapping The score vector of the i-th cluster center among the cluster centers :

[0029] in, are learnable scoring weights, is a learnable rating bias; 3.3) Normalize the score vector Softmax to the soft membership weight of the cluster : in, represents the normalization function, Zk represents the score vector of the k-th cluster center, express Cluster centers j The degree of membership; 3.4) Using the soft membership weights The weighted summation of the prompt segments corresponding to the cluster centers generates a prompt vector that matches the current modality task: in, is the prompt vector of the i-th layer, is the cluster center matrix, , Indicates the j The prompt fragments corresponding to the cluster centers, L p Indicates the length of the prompt segment, Represents a composite function for scoring, Softmax, and calculating soft membership weights.

[0030] By extracting the key information of the input features and multiplying it with the cluster center matrix, a prompt vector that can reflect the characteristics of the current modality task is generated.

[0031] In step 4, the hint vector is inserted into the self-attention layer using a prefix hint strategy or a pre-hint strategy to expand the feature expression before performing self-attention calculation. Before inserting the self-attention layer, the cluster center matrix, the general subspace representation, and the specific subspace representations for the distortion-free task, the audio-only distortion task, and the video-only distortion task are initialized.

[0032] 4.1) Pro-Prompt strategy: The generated prompt vector of the i-th layer of the current modality task Directly concatenate with the input sequence of the audio-video multimodal language model to form enhanced input data, which is inserted into the query of the self-attention layer. Q ,key K Sum V Then, self-attention calculation is performed.

[0033] .

[0034] 4.2) Pre-Prompt Strategy: When generating a prompt vector, only the prompt vector is generated. Generate keytips separately instead Sum value prompt Two parts, inserted into the key of the self-attention layer K Sum V Then, self-attention calculation is performed.

[0035] .

[0036] The upper right corner of the symbol n for c 、 ad or vd , c represents the distortion-free task, ad represents the audio distortion task only, vd represents only the video distortion task (i.e., the three modality task).

[0037] Step 5) Autoregressively generate the output sequence in the feature decoder and calculate the cross entropy loss between it and the true label.

[0038] The calculation formula of cross entropy loss is as follows: in, is the cross entropy loss, Before t- 1 language model input corresponding to the correct text, f Represents the output sequence generated by the feature decoder autoregression, the p in logp represents the conditional probability, and the summation symbol s Indicates the sequence length of the language model input.

[0039] Step 6) Split the cue vector of each modality task into a universal subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three modality tasks.

[0040] The cue vector of each modality task is decomposed into a universal subspace representation and specific subspace representations of the three modality tasks. The decomposition formula is: , , , , in, represents the universal subspace representation, represents the specific subspace representation of the current modality task n, represents the prompt vector for the current modality task n, r Represents the dimensionality of the low-rank decomposition.

[0041] Step 7) Weighted fusion of cross entropy loss and contrast loss, followed by back propagation, jointly update all cue vectors and parameters of the audio-video multimodal language model.

[0042] The calculation formula of the contrast loss is: in, is the temperature coefficient, represents the similarity between vectors, represents the specific subspace representation of the current modality task n, Represents a specific subspace representation of positive sample tasks that are close to the current modality task, represents the task-specific subspace representation of the remaining negative samples, Represents a collection of modal task types, c represents the distortion-free task, ad represents the audio distortion task only, vd represents only the video distortion task; Represents a set of tasks similar to the current modality task n. For tasks with only audio distortion, this set includes them and tasks without distortion; for tasks with only video distortion, this set includes them and tasks without distortion; for tasks without distortion, this set only includes them.

[0043] The formula for weighted fusion cross entropy loss and contrast loss is:

[0044] in, is the cross entropy loss, is the contrast loss, is the loss weight coefficient.

[0045] Jointly update through optimization algorithms such as stochastic gradient descent 、 、 、 、 and the remaining parameters of the audio-video multimodal language model.

[0046] Step 8) Repeat the above steps until the performance on the validation set converges or the predetermined number of training rounds is reached.

[0047] Application Examples The performance of the language model (i.e., the audio-video multimodal language model of the present invention, hereinafter referred to as the language model) was evaluated on the LRS2 dataset, thereby verifying the significant effectiveness of the method of the present invention (i.e., Example 1). The LRS2 dataset contains multimodal data covering audio and video speech recognition tasks. It contains a total of approximately 143,000 video clips with highly aligned audio and video modalities and accurate text transcriptions. The clip lengths range from 1 to 6 seconds. It is primarily used for evaluating and training speech recognition models in multimodal scenarios.

[0048] An objective evaluation method was used to assess the performance of models trained using the method of the present invention. Specifically, the present invention used the word error rate (WER) as the primary evaluation metric. In experiments, based on the AV-HuBERT model (with 300 million pre-trained model parameters), the present invention tested its performance under different modal distortion scenarios, including complete distortion, audio-only distortion, and video distortion. Compared with traditional fine-tuning methods, the present invention significantly reduced the word error rate in various scenarios, while the model parameter overhead accounted for less than 4% of the total model parameters. Table 1 below shows the WER scores of the present invention on the LRS2 dataset.

[0049] Table 1: WER score results on the LRS2 dataset

[0050] (Note: Represents the video distortion and audio distortion data in the entire dataset, accounting for / 2; Represents the remaining (1- ) Each fragment in the data has a ratio of The part is replaced by distorted data, and the underlined numbers represent the best data under the same model and training set length conditions) The results show that the method proposed in the present invention performs significantly better than the existing most advanced methods in three modal tasks: complete distortion, audio distortion only, and video distortion only. In particular, the accuracy is more significantly improved when the proportion of distorted data is large, proving that the method proposed in the present invention improves the model's ability to perform speech recognition on modally distorted data.

[0051] Example 2 This embodiment provides a modal distortion speech recognition system, such as Figure 2 As shown, the method for implementing the modal distortion speech recognition method described in Example 1 is composed of a loading unit, an audio-visual feature acquisition unit, a prompt vector generation unit, a prompt strategy insertion unit, a cross-entropy loss calculation unit, a contrast loss calculation unit, a total loss calculation unit and a loop unit.

[0052] Loading unit: Loads the pre-trained audio encoder, video encoder, audio and video feature fusion module, feature encoder and feature decoder into the audio-video multimodal language model, and loads the alignment parameters; used to implement the function of step 1) in Example 1, which will not be repeated here.

[0053] Audio-visual feature acquisition unit: uses an audio encoder and a video encoder to simultaneously encode the audio and video input of the audio-video multimodal language model, and fuses them in the audio and video feature fusion module to obtain temporal features. The temporal features are then encoded by the feature encoder to obtain audio-visual features; it is used to implement the function of step 2) in Example 1 and will not be repeated here.

[0054] Hint vector generation unit: used to calculate the soft distribution of audiovisual features through the cluster weight extraction network, and generate the hint vector of each layer in combination with the cluster center matrix; used to implement the function of step 3) in embodiment 1, which will not be repeated here.

[0055] Hint strategy insertion unit: Use prefix hint strategy or pre-hint strategy to insert hint vector into self-attention layer, and then perform self-attention calculation; initialize cluster center matrix, general subspace representation and specific subspace representation under distortion-free task, audio distortion-only task and video distortion-only task before inserting into the self-attention layer; used to implement the function of step 4) in embodiment 1, which will not be repeated here.

[0056] Cross-entropy loss calculation unit: used to autoregressively generate an output sequence in the feature decoder and calculate the cross-entropy loss between it and the true label; used to implement the function of step 5) in Example 1, which will not be repeated here.

[0057] Contrastive loss calculation unit: used to split the prompt vector of each task into a universal subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three tasks; used to implement the function of step 6) in Example 1, which will not be repeated here.

[0058] Total loss calculation unit: used to weight the fusion of cross entropy loss and contrast loss, and then perform backpropagation to jointly update all prompt vectors and the remaining parameters of the audio-video multimodal language model; used to implement the function of step 7) in Example 1, which will not be repeated here.

[0059] Loop unit: cyclically executes the above units until the performance on the validation set converges or reaches a predetermined number of training rounds; it is used to implement the function of step 8) in Example 1 and will not be repeated here.

[0060] It should be noted that the various units in the above-mentioned modal distortion speech recognition system can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units. For the specific definition of a modal distortion speech recognition system, please refer to the definition of the modal distortion speech recognition system method (i.e., Example 1) above. The two have the same functions and effects and will not be repeated here.

[0061] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structures or equivalent process changes made using the contents of the present invention's description and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A modal distortion speech recognition method, characterized in that: Including steps: 1) Load the pre-trained audio encoder, video encoder, audio and video feature fusion module, feature encoder, and feature decoder into the audio-video multimodal language model, and load the alignment parameters; 2) Using an audio encoder and a video encoder to simultaneously encode the audio and video inputs of the audio-video multimodal language model, and fusing them in an audio-video feature fusion module to obtain temporal features. The temporal features are then encoded by a feature encoder to obtain audio-visual features. 3) Calculate the soft distribution of audiovisual features through the cluster weight extraction network, and generate the prompt vector of each layer in combination with the cluster center matrix; 4) Using a prefix cue strategy or a pre-cue strategy, the cue vector is inserted into the self-attention layer before performing self-attention calculations; before inserting the cue vector into the self-attention layer, the cluster center matrix, the general subspace representation, and the specific subspace representations for the distortion-free task, the audio-only distortion task, and the video-only distortion task are initialized. 5) Autoregressively generate the output sequence in the feature decoder and calculate the cross entropy loss between it and the true label; 6) Split the cue vector of each task into a universal subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three tasks; 7) Weighted fusion of cross entropy loss and contrastive loss, followed by backpropagation to jointly update all cue vectors and the remaining parameters of the audio-video multimodal language model; 8) Repeat the above steps until the performance on the validation set converges or the predetermined number of training rounds is reached.

2. The modal distortion speech recognition method according to claim 1, characterized in that: Step 3) specifically includes: 3.1) Cluster the audiovisual features in the time series dimension to obtain the membership distribution of the audiovisual features : in, Represents clustering operation; Indicates audiovisual characteristics; represents the field of real numbers; T Indicates the length of the time series; D Represents the dimension of the feature; 3.2) Obtain the pair through linear mapping The score vector of the i-th cluster center among the cluster centers : in, are learnable scoring weights, is a learnable rating bias; 3.3) Normalize the score vector Softmax to the soft membership weight of the cluster : in, represents the normalization function, Z k represents the score vector of the k-th cluster center, express Cluster centers j The degree of membership; 3.4) Using the soft membership weights The weighted summation of the prompt segments corresponding to the cluster centers generates a prompt vector that matches the current modality task: in, is the prompt vector of the i-th layer, is the cluster center matrix, , Indicates the j The prompt fragments corresponding to the cluster centers, L p Indicates the length of the prompt segment, Represents a composite function for scoring, Softmax, and calculating soft membership weights.

3. The modal distortion speech recognition method according to claim 1, characterized in that: The prompting strategy in step 4) includes: 4.1) Pre-hint strategy: The generated hint vector is directly concatenated with the input sequence of the audio-video multimodal language model to form enhanced input data, which is inserted into the query, key, and value of the self-attention layer.

4. The modal distortion speech recognition method according to claim 1, characterized in that: The prompting strategy in step 4) includes: 4.2) Prefix Hint Strategy: When generating the hint vector, instead of just generating the hint vector, we generate both the key hint and the value hint, which are then inserted into the key and value of the self-attention layer respectively.

5. The modal distortion speech recognition method according to claim 2, characterized in that: In step 5), the calculation formula of cross entropy loss is as follows: in, is the cross entropy loss, Before t- 1 language model input corresponding to the correct text, f Represents the output sequence generated by the feature decoder autoregression, the p in logp represents the conditional probability, and the summation symbol s Indicates the sequence length of the language model input.

6. The modal distortion speech recognition method according to claim 1, characterized in that: In step 6), the cue vector of each modality task is decomposed into a universal subspace representation and three task-specific subspace representations. The decomposition formula is: ， in, represents the universal subspace representation, represents the specific subspace representation of the current modality task n, Represents the prompt vector for the current modality task n.

7. The modal distortion speech recognition method according to claim 1, characterized in that: The calculation formula of the contrast loss is: in, is the temperature coefficient, represents the similarity between vectors, represents the specific subspace representation of the current modality task n, Represents a specific subspace representation of positive sample tasks that are close to the current modality task, represents the task-specific subspace representation of the remaining negative samples, Represents a collection of modal task types, c represents the distortion-free task, ad represents the audio distortion task only, vd represents only the video distortion task; Represents a set of tasks that are similar to the current modal task n.

8. The modal distortion speech recognition method according to claim 7, characterized in that: In step 7), the formula for weighted fusion cross entropy loss and contrast loss is: in, is the cross entropy loss, is the contrast loss, is the loss weight coefficient.

9. The modal distortion speech recognition method according to claim 8, characterized in that: All cue vectors and parameters of the audio-visual multimodal language model are jointly updated via a stochastic gradient descent optimization algorithm.

10. A modal distortion speech recognition system, used to implement the modal distortion speech recognition method according to any one of claims 1 to 9, characterized in that: include: Loading unit: loads the pre-trained audio encoder, video encoder, audio and video feature fusion module, feature encoder, and feature decoder into the audio-video multimodal language model, and loads the alignment parameters; Audio-visual feature acquisition unit: This unit uses an audio encoder and a video encoder to simultaneously encode the audio and video inputs of the audio-video multimodal language model. These are then fused in the audio-video feature fusion module to obtain temporal features, which are then encoded by the feature encoder to obtain audio-visual features. Hint vector generation unit: used to calculate the soft distribution of audiovisual features through the cluster weight extraction network, and generate the hint vector of each layer in combination with the cluster center matrix; Hint strategy insertion unit: Use prefix hint strategy or pre-hint strategy to insert hint vector into the self-attention layer, and then perform self-attention calculation; before inserting into the self-attention layer, initialize the cluster center matrix, general subspace representation, and specific subspace representations under the distortion-free task, audio distortion-only task, and video distortion-only task; Cross-entropy loss calculation unit: used to autoregressively generate the output sequence in the feature decoder and calculate the cross-entropy loss between it and the true label; Contrastive loss calculation unit: used to split the prompt vector of each task into a universal subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three tasks; Total loss calculation unit: used to weight the fusion of cross entropy loss and contrast loss, and then perform backpropagation to jointly update all prompt vectors and the remaining parameters of the audio-video multimodal language model; Loop unit: The above units are executed cyclically until the performance on the validation set converges or the predetermined number of training rounds is reached.

Images