A method and system for modal distortion speech recognition

Abstract

The application discloses a modal distortion speech recognition method and system. The application calculates the soft assignment of audio-visual features through a cluster weight extraction network, and generates a prompt vector of each layer in combination with a clustering center matrix; adopts a prefix prompt strategy or a preposition prompt strategy to insert the prompt vector into a self-attention layer; generates an output sequence in a feature decoder in a self-recurrence manner, and calculates the cross-entropy loss between the output sequence and a real label; splits the prompt vector of each task into a general subspace representation and a specific subspace representation, and calculates the contrast loss of the specific subspace representations of the three tasks; and fuses the cross-entropy loss and the contrast loss in a weighted manner, and then performs back propagation to jointly update the parameters of all prompt vectors and language models. The application reduces the difference between the modal missing prompt and the non-distortion prompt, and increases the difference between the prompts of different modal tasks, thereby improving the recognition effect of the audio-video multi-modal language model under different modal tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal learning, in particular to a modal distortion speech recognition method and system. BACKGROUND

[0002] Audio-visual speech recognition utilizes the synergy of speech signals and lip video to generate natural language text, and shows high accuracy in multi-modal learning. However, in actual scenarios, due to recording equipment and environmental restrictions, data is often affected by modal distortion (such as audio-video asynchrony, audio noise or video blur, etc.), which seriously reduces the performance of existing audio-visual speech recognition models.

[0003] In recent years, prompt learning as a lightweight fine-tuning method has achieved remarkable success in natural language processing. By designing input prompts for pre-trained models, it can quickly adapt to specific task requirements while preserving the pre-training knowledge of the model. In the field of multi-modal learning, the application of prompt learning is still in its infancy. Traditional prompt learning methods mainly focus on single-modal scenarios, and how to design an efficient prompt mechanism to cope with the complexity of multi-modal data under modal distortion conditions is still a problem to be solved. SUMMARY

[0004] The technical problem to be solved by the present application is to overcome the defects of the prior art, and to provide a modal distortion speech recognition method and system. Based on a new framework of contrastive decomposition and clustering prompts, the robustness of the audio-video multi-modal language model under multi-modal distortion conditions is enhanced through prompt learning, the prompt features of each modal distortion are constrained by contrastive loss, the difference between the modal missing prompt and the distortion-free prompt is reduced, and the difference between the prompts of different modal distortion tasks is increased, so as to significantly improve the recognition effect of the audio-video multi-modal language model under different modal distortion tasks.

[0005] To this end, the present application adopts the following technical solution: a modal distortion speech recognition method, comprising the steps of:

[0006] 1) loading a pre-trained audio encoder, video encoder, audio-video feature fusion module, feature encoder and feature decoder in the audio-video multi-modal language model, and loading the alignment parameters;

[0007] 2) using the audio encoder and the video encoder to simultaneously encode the audio and the video of the input of the audio-video multi-modal language model, and fusing in the audio-video feature fusion module to obtain the time sequence feature, which is then encoded by the feature encoder to obtain the audio-visual feature;

[0008] 3) calculating the soft assignment of the audio-visual feature through a cluster weight extraction network, and generating a prompt vector for each layer in combination with a clustering center matrix;

[0009] 4) The prompt vectors are inserted into the self-attention layer using the prefix prompt strategy or the prefix prompt strategy, and then the self-attention calculation is performed; the cluster center matrix, the general subspace representation, and the specific subspace representation under the distortion-free task, the audio-only distortion task, and the video-only distortion task are initialized before the self-attention layer is inserted;

[0010] 5) The output sequence is generated in the feature decoder in an autoregressive manner, and the cross-entropy loss between the output sequence and the true label is calculated;

[0011] 6) The prompt vectors of each task are split into general subspace representations and specific subspace representations, and the specific subspace representations of the three tasks are calculated by contrastive loss;

[0012] 7) The cross-entropy loss and the contrastive loss are weighted and fused, and then backpropagation is performed to jointly update all the prompt vectors and the remaining parameters of the audio-video multi-modal language model;

[0013] 8) The above steps are repeatedly performed until the performance converges on the validation set or the predetermined number of training rounds is reached.

[0014] The present application uses clustering to process inputs under various modal tasks, generates different prompt vectors inserted into the input or intermediate data of the self-attention layer, and guides the audio-video multi-modal language model to adapt to different types of modal tasks. In order to further optimize the prompt, the present application uses low-rank decomposition to decompose the prompt vector into general subspace representation and specific subspace representation, and uses contrastive loss to constrain the prompt features of each modal task. The present application aims to improve the ability of the audio-video multi-modal language model to perform speech recognition under multiple modal tasks.

[0015] Further, step 3) specifically includes:

[0016] 3.1) Clustering the audio-visual features in the time sequence dimension to obtain the membership distribution of the audio-visual features :

[0017]

[0018] wherein, represents the clustering operation; represents the audio-visual features; represents the real number field; T represents the time sequence length; D represents the dimension of the feature;

[0019] 3.2) obtaining the score vector of the i-th cluster center in the cluster center by linear mapping :

[0020] ​

[0021] in, are learnable scoring weights, is a learnable rating bias;

[0022] 3.3) Normalize the score vector Softmax to the soft membership weight of the cluster :

[0023]

[0024] in, represents the normalization function, Z k represents the score vector of the k-th cluster center, express Cluster centers j The degree of membership;

[0025] 3.4) Using the soft membership weights The weighted summation of the prompt segments corresponding to the cluster centers generates a prompt vector that matches the current modality task:

[0026]

[0027] in, is the prompt vector of the i-th layer, is the cluster center matrix, , Indicates the j The prompt fragments corresponding to the cluster centers, L p Indicates the length of the prompt segment, Represents a composite function for scoring, Softmax, and calculating soft membership weights.

[0028] Furthermore, the prompt strategy in step 4) includes:

[0029] 4.1) Pre-hint strategy: The generated hint vector is directly concatenated with the input sequence of the audio-video multimodal language model to form enhanced input data, which is inserted into the query, key, and value of the self-attention layer.

[0030] Furthermore, the prompt strategy in step 4) includes:

[0031] 4.2) Prefix Hint Strategy: When generating the hint vector, instead of just generating the hint vector, we generate both the key hint and the value hint, which are then inserted into the key and value of the self-attention layer respectively.

[0032] Furthermore, in step 5), the calculation formula of the cross entropy loss is as follows:

[0033]

[0034] wherein, is the cross-entropy loss, denotes the pre t- 1 language model input corresponding to the correct text, f denotes the output sequence generated by the feature decoder autoregressively, p in logp denotes the conditional probability, and the summation symbol is s denotes the sequence length of the language model input.

[0035] Further, in the step 6), the prompt vector of each task is decomposed into a general subspace representation and a specific subspace representation of the three tasks, and the decomposition formula is:

[0036] ,

[0037] wherein, denotes the general subspace representation, denotes the specific subspace representation of the current modal task n, denotes the prompt vector of the current modal task n.

[0038] Further, the calculation formula of the contrastive loss is:

[0039]

[0040] wherein, is the temperature coefficient, denotes the similarity between vectors, denotes the specific subspace representation of the current modal task n, denotes the specific subspace representation of the positive sample task similar to the current modal task, denotes the specific subspace representation of the remaining negative sample task, denotes the set of modal task types, c denotes the undistorted task, ad denotes the only audio distorted task, vd denotes the only video distorted task; denotes the set of tasks similar to the current modal task n.

[0041] Further, in the step 7), the formula for weighted fusion of the cross-entropy loss and the contrastive loss is:

[0042]

[0043] wherein, is the cross-entropy loss, is the contrastive loss, is the loss weight coefficient.

[0044] Further, all prompt vectors and the rest of the audio-video multi-modal language model are updated jointly by a stochastic gradient descent optimization algorithm.

[0045] The application also provides a modal distortion speech recognition system for implementing the above modal distortion speech recognition method, comprising:

[0046] A loading unit: pre-trained audio encoder, video encoder, audio-video feature fusion module, feature encoder and feature decoder are loaded in the audio-video multi-modal language model, and alignment parameters are loaded;

[0047] An audio-visual feature acquisition unit: audio and video encoding of the input of the audio-video multi-modal language model is performed simultaneously by using the audio encoder and the video encoder, and the time sequence features are obtained by fusion in the audio-video feature fusion module, and the audio-visual features are obtained by encoding the time sequence features by the feature encoder;

[0048] A prompt vector generation unit: used for calculating the soft assignment of the audio-visual features by the cluster weight extraction network, and generating the prompt vectors of each layer in combination with the cluster center matrix;

[0049] A prompt strategy insertion unit: the prompt vectors are inserted into the self-attention layer by using the prefix prompt strategy or the preposition prompt strategy, and then the self-attention calculation is performed; the cluster center matrix, the general subspace representation and the specific subspace representation under the distortion-free task, the only audio distortion task and the only video distortion task are initialized before the insertion of the self-attention layer;

[0050] A cross-entropy loss calculation unit: used for generating the output sequence in the feature decoder by autoregression, and calculating the cross-entropy loss between the output sequence and the real label;

[0051] A contrastive loss calculation unit: used for splitting the prompt vectors of each task into the general subspace representation and the specific subspace representation, and performing the contrastive loss calculation on the specific subspace representations of the three tasks;

[0052] A total loss calculation unit: used for weightedly fusing the cross-entropy loss and the contrastive loss, and then performing the back propagation to update all the prompt vectors and the rest of the audio-video multi-modal language model jointly;

[0053] A loop unit: the above units are executed in a loop until the performance converges on the validation set or the predetermined training round is reached.

[0054] The beneficial effects of the present invention are as follows: the present invention enhances the robustness of the audio-video multimodal language model in multimodal tasks through prompt learning, constrains the prompt features of each modal task through contrast loss, reduces the difference between modality missing prompts and distortion-free prompts, and increases the difference between prompts of different modal tasks, thereby significantly improving the recognition effect of the audio-video multimodal language model in different modal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a modal distortion speech recognition method of the present invention;

[0056] Figure 2 This is a composition diagram of a modal distortion speech recognition system of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific implementation methods.

[0058] Example 1

[0059] like Figure 1 As shown in FIG, the present invention is a method for modal distortion speech recognition based on contrast decomposition and clustering prompts, such as Figure 1 As shown, the steps are as follows:

[0060] Step 1) Load the pre-trained audio encoder into the audio-video multimodal language model , video encoder , audio and video feature fusion module AVFusion, feature encoder and feature decoder EncoderLayer, and load their alignment parameters.

[0061] In step 2, the audio encoder and video encoder are used to simultaneously encode the audio and video input of the audio-video multimodal language model, and the temporal features are obtained by fusion in the audio and video feature fusion module. The temporal features are then encoded by the feature encoder to obtain audio-visual features.

[0062] Encode audio and video into features separately and fuse them:

[0063]

[0064] in, represents the time series features after fusion, Represents audio-visual characteristics, represents the field of real numbers; T Indicates the length of the time series; D Represents the dimension of the feature, Indicates audio input; v Indicates video input.

[0065] Step 3) Calculate the soft distribution of audiovisual features through the cluster weight extraction network, and generate the prompt vector of each layer in combination with the cluster center matrix.

[0066] 3.1) Cluster the audiovisual features in the time series dimension to obtain the membership distribution of the audiovisual features :

[0067]

[0068] in, Represents clustering operation; Indicates audiovisual characteristics; represents the field of real numbers; T Indicates the length of the time series; D Represents the dimension of the feature;

[0069] 3.2) Obtain the pair through linear mapping The score vector of the i-th cluster center among the cluster centers :

[0070]

[0071] in, are learnable scoring weights, is a learnable rating bias;

[0072] 3.3) Normalize the score vector Softmax to the soft membership weight of the cluster :

[0073]

[0074] in, represents the normalization function, Z k represents the score vector of the k-th cluster center, express Cluster centers j The degree of membership;

[0075] 3.4) Using the soft membership weights The weighted summation of the prompt segments corresponding to the cluster centers generates a prompt vector that matches the current modality task:

[0076]

[0077] in, is the prompt vector of the i-th layer, is the cluster center matrix, , Indicates the j The prompt fragments corresponding to the cluster centers, L pdenotes the length of the prompt segment, denotes the composite function for scoring, Softmax and computing soft membership weights.

[0078] By extracting the key information of the input features and multiplying it with the cluster center matrix, a prompt vector that can reflect the characteristics of the current modal task is generated.

[0079] Step 4), use the prefix prompt strategy or pre-prompt strategy to insert the prompt vector into the self-attention layer to expand the feature expression, and then perform self-attention calculation; initialize the cluster center matrix, general subspace representation and specific subspace representation under the distortion-free task, audio-only distortion task and video-only distortion task before inserting the self-attention layer.

[0080] 4.1) Pro-Prompt: the generated prompt vector of the current modal task i-th layer is directly concatenated with the input sequence of the audio-video multi-modal language model to form enhanced input data, which is inserted into the query Q , key K and value V of the self-attention layer, and then self-attention calculation is performed.

[0081] .

[0082] 4.2) Pre-Prompt: when generating the prompt vector, instead of generating the prompt vector , generate key prompt and value prompt two parts respectively, which are inserted into the key K and value V of the self-attention layer, and then self-attention calculation is performed.

[0083] .

[0084] The upper right corner of the symbol n is c , ad or vd , c denotes the distortion-free task, ad denotes the audio-only distortion task, vd denotes the video-only distortion task (i.e. three modal tasks).

[0085] Step 5), generate the output sequence in the feature decoder autoregressively, and calculate the cross-entropy loss between it and the true label.

[0086] The calculation formula of cross-entropy loss is as follows:

[0087]

[0088] wherein, is the cross-entropy loss, denotes the pre t- 1 correct text corresponding to the language model input, f denotes the output sequence generated by the feature decoder autoregressively, p in logp denotes the conditional probability, and the summation symbol is s denotes the sequence length of the language model input.

[0089] Step 6), split the prompt vector of each modal task into a general subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three modal tasks.

[0090] The prompt vector of each modal task is decomposed into a general subspace representation and a specific subspace representation of the three modal tasks, and the decomposition formula is:

[0091] , , , ,

[0092] wherein, denotes the general subspace representation, denotes the specific subspace representation of the current modal task n, denotes the prompt vector of the current modal task n, r denotes the dimension of the low-rank decomposition.

[0093] Step 7), weighted fusion of cross-entropy loss and contrastive loss, followed by back propagation, and joint update of all prompt vectors and audio-video multi-modal language model parameters.

[0094] The calculation formula of the contrastive loss is:

[0095]

[0096] wherein, is the temperature coefficient, denotes the similarity between vectors, denotes the specific subspace representation of the current modal task n, denotes the specific subspace representation of the positive sample task similar to the current modal task, denotes the specific subspace representation of the remaining negative sample task, denotes the set of modal task types, c denotes the distortion-free task, ad denotes the only audio distortion task, vd denotes the only video distortion task; denotes the set of tasks similar to the current modality task n, for audio-only distortion tasks, this set includes themselves and the no-distortion task; for video-only distortion tasks, this set includes themselves and the no-distortion task; for no-distortion tasks, this set only includes themselves.

[0097] The formula for weighting and fusing the cross-entropy loss and the contrastive loss is:

[0098]

[0099] wherein, is the cross-entropy loss, is the contrastive loss, is the loss weight coefficient.

[0100] Through optimization algorithms such as stochastic gradient descent, jointly update , , , , and the rest of the parameters of the audio-video multi-modal language model.

[0101] Step 8), the above steps are executed in a loop until the performance converges on the validation set or the predetermined number of training rounds is reached.

[0102] Application Example

[0103] The performance of the language model (i.e., the audio-video multi-modal language model of the present application, referred to as the language model) is evaluated on the LRS2 dataset, thereby verifying the significant effect of the method (i.e., Example 1) of the present application. The LRS2 dataset contains multi-modal data covering audio-video speech recognition tasks, which contains a total of about 143,000 video clips, with highly aligned audio and video modalities, and with accurate text transcriptions, clip lengths ranging from 1 to 6 seconds, and is mainly used to evaluate and train speech recognition models in multi-modal scenarios.

[0104] Objective evaluation method is used to evaluate the performance of the model trained by the method of the present application. Specifically, the present application uses the word error rate (Word Error Rate, WER) as the main evaluation index. In the experiment, the present application tests the performance in different modal distortion scenarios based on the AV-HuBERT model (the pre-training model has 300 million parameters), including complete distortion, only audio distortion and video distortion. Compared with the traditional fine-tuning method, the method of the present application significantly reduces the word error rate in multiple scenarios, while the model parameter overhead is less than 4% of the total model parameters. Table 1 below shows the WER score of the present application on the LRS2 dataset.

[0105] Table 1: WER score results on the LRS2 dataset

[0106]

[0107] (Note: The data representing video distortion and audio distortion in the entire data set accounts for / 2 respectively; The remaining (1- ) data is replaced by a portion of The underlined numbers represent the best data under the same model and training set length conditions)

[0108] The results show that the method proposed in the application performs significantly better than the most advanced method in the three modal tasks of complete distortion, only audio distortion and only video distortion, especially when the proportion of distorted data is larger, the accuracy is significantly improved, which proves that the method proposed in the application improves the ability of the model to recognize speech from modal distorted data.

[0109] Embodiment 2

[0110] The embodiment provides a modal distortion speech recognition system, as shown in Figure 2 , for implementing the modal distortion speech recognition method described in embodiment 1, which is composed of a loading unit, an audio-visual feature acquisition unit, a prompt vector generation unit, a prompt strategy insertion unit, a cross-entropy loss calculation unit, a contrast loss calculation unit, a total loss calculation unit and a loop unit.

[0111] Loading unit: load the pre-trained audio encoder, video encoder, audio-video feature fusion module, feature encoder and feature decoder in the audio-video multi-modal language model, and load the alignment parameters; for realizing the function of step 1) in embodiment 1, which will not be repeated here.

[0112] Audio-visual feature acquisition unit: use the audio encoder and the video encoder to simultaneously encode the input of the audio-video multi-modal language model, and fuse in the audio-video feature fusion module to obtain the time sequence feature, which is then encoded by the feature encoder to obtain the audio-visual feature; for realizing the function of step 2) in embodiment 1, which will not be repeated here.

[0113] Prompt vector generation unit: for calculating the soft assignment of audio-visual features through the cluster weight extraction network, and generating the prompt vector of each layer combined with the clustering center matrix; for realizing the function of step 3) in embodiment 1, which will not be repeated here.

[0114] Prompt strategy insertion unit: insert prompt vectors into self-attention layers using prefix prompt strategy or prefix prompt strategy, and then perform self-attention calculation; initialize cluster center matrix, general subspace representation, and specific subspace representation under lossless task, audio-only distortion task, and video-only distortion task before inserting self-attention layers; used to implement the function of step 4) in embodiment 1, and details are not repeated here.

[0115] Cross-entropy loss calculation unit: used to generate output sequences in the feature decoder in an autoregressive manner, and calculate the cross-entropy loss between the output sequences and the real labels; used to implement the function of step 5) in embodiment 1, and details are not repeated here.

[0116] Contrastive loss calculation unit: used to split the prompt vectors of each task into general subspace representation and specific subspace representation, and calculate the contrastive loss of the specific subspace representation of the three tasks; used to implement the function of step 6) in embodiment 1, and details are not repeated here.

[0117] Total loss calculation unit: used to weight and fuse the cross-entropy loss and the contrastive loss, and then perform back propagation to jointly update all prompt vectors and the remaining parameters of the audio-video multi-modal language model; used to implement the function of step 7) in embodiment 1, and details are not repeated here.

[0118] Loop unit: loop the above units until the performance converges on the validation set or the predetermined number of training rounds is reached; used to implement the function of step 8) in embodiment 1, and details are not repeated here.

[0119] It should be noted that each unit in the above modal distortion speech recognition system can be realized by software, hardware, and combinations thereof, in whole or in part. Each unit described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each unit. For specific limitations of a modal distortion speech recognition system, see the limitations of the modal distortion speech recognition system method (i.e., embodiment 1) in the above, both of which have the same functions and effects, and details are not repeated here.

[0120] The above only describes the embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A modal distortion speech recognition method, characterized in that: Including steps: 1) Load the pre-trained audio encoder, video encoder, audio and video feature fusion module, feature encoder, and feature decoder into the audio-video multimodal language model, and load the alignment parameters; 2) Using an audio encoder and a video encoder to simultaneously encode the audio and video inputs of the audio-video multimodal language model, and fusing them in an audio-video feature fusion module to obtain temporal features. The temporal features are then encoded by a feature encoder to obtain audio-visual features. 3) Calculate the soft distribution of audiovisual features through the cluster weight extraction network, and combine it with the cluster center matrix to generate the prompt vector of each layer; 4) Using a prefix cue strategy or a pre-cue strategy, the cue vector is inserted into the self-attention layer before performing self-attention calculations; before inserting the cue vector into the self-attention layer, the cluster center matrix, the general subspace representation, and the specific subspace representations for the distortion-free task, the audio-only distortion task, and the video-only distortion task are initialized. 5) Autoregressively generate the output sequence in the feature decoder and calculate the cross entropy loss between it and the true label; 6) Split the cue vector of each task into a universal subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three tasks; 7) Weighted fusion of cross entropy loss and contrastive loss, followed by backpropagation to jointly update all cue vectors and the remaining parameters of the audio-video multimodal language model; 8) Repeat the above steps until the performance on the validation set converges or the predetermined number of training rounds is reached.

2. The modal distortion speech recognition method according to claim 1, characterized in that: Step 3) specifically includes: 3.1) Cluster the audiovisual features in the time series dimension to obtain the membership distribution of the audiovisual features : in, Represents clustering operation; Indicates audiovisual characteristics; represents the field of real numbers; T Indicates the length of the time series; D Represents the dimension of the feature; 3.2) Obtain the pair through linear mapping The score vector of the i-th cluster center among the cluster centers : in, are learnable scoring weights, is a learnable rating bias; 3.3) Normalize the score vector Softmax to the soft membership weight of the cluster : in, represents the normalization function, Z k represents the score vector of the k-th cluster center, express Cluster centers j The degree of membership; 3.4) Using the soft membership weights The weighted summation of the prompt segments corresponding to the cluster centers generates a prompt vector that matches the current modality task: in, is the prompt vector of the i-th layer, is the cluster center matrix, , Indicates the j The prompt fragments corresponding to the cluster centers, L p Indicates the length of the prompt segment, Represents a composite function for scoring, Softmax, and calculating soft membership weights.

3. The modal distortion speech recognition method according to claim 1, characterized in that: The prompting strategy in step 4) includes: 4.1) Pre-hint strategy: The generated hint vector is directly concatenated with the input sequence of the audio-video multimodal language model to form enhanced input data, which is inserted into the query, key, and value of the self-attention layer.

4. The modal distortion speech recognition method according to claim 1, characterized in that: The prompting strategy in step 4) includes: 4.2) Prefix Hint Strategy: When generating the hint vector, instead of just generating the hint vector, we generate both the key hint and the value hint, which are then inserted into the key and value of the self-attention layer respectively.

5. The modal distortion speech recognition method according to claim 2, characterized in that: In step 5), the calculation formula of cross entropy loss is as follows: in, is the cross entropy loss, Before t- 1 language model input corresponding to the correct text, f Represents the output sequence generated by the feature decoder autoregression, the p in logp represents the conditional probability, and the summation symbol s Indicates the sequence length of the language model input.

6. The modal distortion speech recognition method according to claim 1, characterized in that: In step 6), the cue vector of each modality task is decomposed into a universal subspace representation and three task-specific subspace representations. The decomposition formula is: ， in, represents the universal subspace representation, represents the specific subspace representation of the current modality task n, Represents the prompt vector for the current modality task n.

7. The modal distortion speech recognition method according to claim 1, characterized in that: The calculation formula of the contrast loss is: in, is the temperature coefficient, represents the similarity between vectors, represents the specific subspace representation of the current modality task n, Represents a specific subspace representation of positive sample tasks that are close to the current modality task, represents the task-specific subspace representation of the remaining negative samples, Represents a collection of modal task types, c represents the distortion-free task, ad represents the audio distortion task only, vd represents only the video distortion task; Represents a set of tasks that are similar to the current modal task n.

8. The modal distortion speech recognition method according to claim 7, characterized in that: In step 7), the formula for weighted fusion cross entropy loss and contrast loss is: in, is the cross entropy loss, is the contrast loss, is the loss weight coefficient.

9. The modal distortion speech recognition method according to claim 8, characterized in that: All cue vectors and parameters of the audio-visual multimodal language model are jointly updated via a stochastic gradient descent optimization algorithm.

10. A modal distortion speech recognition system, used to implement the modal distortion speech recognition method according to any one of claims 1 to 9, characterized in that: include: Loading unit: loads the pre-trained audio encoder, video encoder, audio and video feature fusion module, feature encoder, and feature decoder into the audio-video multimodal language model, and loads the alignment parameters; Audio-visual feature acquisition unit: This unit uses an audio encoder and a video encoder to simultaneously encode the audio and video inputs of the audio-video multimodal language model. These are then fused in the audio-video feature fusion module to obtain temporal features, which are then encoded by the feature encoder to obtain audio-visual features. Hint vector generation unit: used to calculate the soft distribution of audiovisual features through the cluster weight extraction network, and generate the hint vector of each layer in combination with the cluster center matrix; Hint strategy insertion unit: Use prefix hint strategy or pre-hint strategy to insert hint vector into the self-attention layer, and then perform self-attention calculation; before inserting into the self-attention layer, initialize the cluster center matrix, general subspace representation, and specific subspace representations under the distortion-free task, audio distortion-only task, and video distortion-only task; Cross-entropy loss calculation unit: used to autoregressively generate the output sequence in the feature decoder and calculate the cross-entropy loss between it and the true label; Contrastive loss calculation unit: used to split the prompt vector of each task into a universal subspace representation and a specific subspace representation, and perform contrastive loss calculation on the specific subspace representations of the three tasks; Total loss calculation unit: used to weight the fusion of cross entropy loss and contrast loss, and then perform backpropagation to jointly update all prompt vectors and the remaining parameters of the audio-video multimodal language model; Loop unit: The above units are executed cyclically until the performance on the validation set converges or the predetermined number of training rounds is reached.

Images