Visual language translation method and system based on contrastive learning and word granularity weight

Abstract

The application discloses a visual language translation method and system based on contrast learning and word granularity weight, and belongs to the field of time alignment visual language translation. Lip or finger video embedding features and text embedding features of a source domain are extracted; the video embedding features are encoded first, then interacted with the text embedding features through a multi-head attention mechanism, and word probability distribution is generated through decoding; a task-based cross entropy loss function item is used for preliminary training; according to the decoded attention vector, the word granularity diversity weight of the word is calculated; the source domain is randomly divided into a meta training set and a meta test set, a contrast limited meta learning training strategy is adopted, the learning direction of the model is controlled through the diversity perception weight, and the encoder and the decoder with generalization ability are trained. The visual language translation task of an unseen person is completed by using the trained visual encoder and the cross-modal decoder. The application improves the generalization ability to out-of-domain speakers and effectively improves the visual language translation effect.

Description

Technical Field

[0001] This invention relates to the field of temporally aligned visual language translation, and more particularly to a visual language translation method and system based on contrastive learning and word granularity weighting. Background Technology

[0002] Temporally aligned visual language translation aims to translate visual content presented by a speaker into natural language text, representing an interdisciplinary field between computer vision and natural language processing. Specifically, this field includes important artificial intelligence tasks such as lip-reading and finger-reading, using automated visual language translators to help hearing-impaired individuals communicate with others. In lip-reading, spoken content is translated based on the speaker's lip movements; in finger-reading, the text sequence is translated based on the fine-grained hand gestures of the finger reader. A common feature of both lip-reading and finger-reading is that the visual content and the translated natural language text are temporally aligned.

[0003] Existing technologies primarily utilize autoregressive methods or connection-based temporal classification to generate word sequences. However, due to the diverse expression habits of different interpreters on specific words prone to ambiguity, these methods perform poorly in real-world scenarios. Furthermore, when data resources are limited or annotation costs are high, these differences between interpreters become even more pronounced. Ideally, an applicable temporally aligned visual language translation system should possess excellent generalization capabilities and deliver high translation accuracy even on unseen interpreters.

[0004] Currently, breakthroughs have been made in domain generalization tasks, which aim to train a model from a source domain with limited data that can directly generalize to an unseen target domain. Existing techniques mainly fall into three categories: representation learning, data manipulation, and learning strategies. Given the need for time-aligned visual language translation tasks to generalize models across speakers, utilizing methods from domain generalization tasks while combining them with the specific characteristics of visual language translation can effectively improve translation accuracy. Summary of the Invention

[0005] The purpose of this invention is to enhance the generalization ability of temporally aligned visual language translation systems on out-of-domain speakers, so as to simultaneously overcome inherent ambiguities between words and maintain inter-class semantic relationships, and improve the domain independence of the model. This invention provides a visual language translation method and system based on contrastive learning and word granular weighting to translate lip-reading and finger-reading videos into natural language text.

[0006] The specific technical solution adopted in this invention is as follows:

[0007] In a first aspect, this invention proposes a visual language translation method based on contrastive learning and word granularity weighting, comprising the following steps:

[0008] 1) Extract lip-reading or finger-reading video embedding features from the source domain and obtain natural language text embedding features;

[0009] 2) Encode the lip-reading or finger-reading video embedding features using a visual encoder based on a multi-head attention mechanism to obtain the encoded visual features;

[0010] 3) For the encoded visual features, the multi-head attention mechanism is used in the cross-modal decoder to interact with the natural language text embedding features, decode and generate word probability distributions, and obtain the task-based cross-entropy loss function term to perform preliminary training on the visual encoder and cross-modal decoder.

[0011] 4) Based on the decoded attention vectors obtained from the initially trained visual encoder and cross-modal decoder, calculate the word granularity diversity weights and update the cross-entropy loss function term in step 3) to obtain the cross-entropy loss term with the effect of word granularity diversity weights.

[0012] 5) The source domain is randomly divided into a meta-training set and a meta-test set. In the meta-training stage, the parameters of the visual encoder and cross-modal decoder are updated using the cross-entropy loss term with word granularity diversity weights. In the meta-testing stage, the global and local contrastive learning loss function terms are obtained using the attention vector calculated in the cross-modal decoder after the meta-testing stage. Combined with the cross-entropy loss term with word granularity diversity weights, the parameters of the visual encoder and cross-modal decoder are updated again to obtain the trained visual encoder and cross-modal decoder.

[0013] 6) Obtain the lip-reading or finger-reading video embedding features to be translated in the target domain, and use the trained visual encoder and cross-modal decoder to complete the visual language translation task for the unseen speaker.

[0014] Furthermore, the lip-reading or finger-reading videos from the source domain are time-aligned with the corresponding natural language text.

[0015] Furthermore, the visual encoder is a multi-layer structure, with each layer consisting of a self-attention layer, a feedforward neural network, residual connections, and layer normalization operations stacked together. It is used to encode the embedded features of lip reading or finger reading videos to generate an encoded video feature matrix.

[0016] Furthermore, the cross-modal decoder has a multi-layer structure, with each layer consisting of a self-attention layer, a mutual attention layer, a feedforward neural network, a residual connection, and a layer normalization operation stacked together. It is used to predict the words at the next time step based on the natural language text embedding features and the encoded video feature matrix.

[0017] Furthermore, step 4) includes:

[0018] 4.1) Input the embedded features of the source domain data into the visual encoder and cross-modal decoder initially trained in step 3), and input the mutual attention layer SA(E) of the last layer of the cross-modal decoder. t ′ ,F ′ ,F ′ The calculation result is denoted as Among them, T t It is the number of words before time step t. This represents the attention vector of the k-th expresser;

[0019] The personalized expression effect of the kth expressor on word c is calculated using the following formula:

[0020]

[0021] in, This indicates that when the r-th word is c, It represents the number of samples with the word label 'c'. Let represent the expression vector of the kth expressor on word c;

[0022] 4.2) Calculate the variance based on the expression vectors of different speakers on the same word. The calculation formula is as follows:

[0023]

[0024] Where K is the number of expressers in the source domain, σ(·) represents the nonlinear activation function, and v c Let represent the diversity weight vector of word c. The complete word-granularity diversity weight matrix is ​​obtained by concatenating the weights of each word, and is represented as follows: T c Indicates the length of the vocabulary;

[0025] 4.3) Based on the word granularity diversity weight matrix obtained in step 4.2), update the cross-entropy loss function term. The calculation formula is as follows:

[0026]

[0027] Here, * represents a positional multiplication operation between vectors at the word level. It is a cross-entropy loss function with word granularity diversity weighting.

[0028] Furthermore, step 5) includes:

[0029] 5.1) The source domain data containing K domains is randomly divided into meta-training sets during each training round. Heyuan Test Set N tr D represents the amount of data in the meta-training set. i This represents the i-th data point in the meta-training set. N te D represents the size of the meta-test set. j This represents the j-th data point in the meta-test set;

[0030] 5.2) During the meta-training phase, calculate the cross-entropy loss term with the influence of word granularity diversity weights, and update the parameters:

[0031]

[0032] Where θ represents all trainable parameters of the visual encoder and cross-modal decoder, α represents the learning rate during the meta-training phase, and θ ′ This indicates the updated parameters. Represents the gradient;

[0033] 5.3) During the meta-testing phase, the data pairs in the meta-test set are updated to θ using the model parameters. ′ The model utilizes the attention vector computed in the decoder to calculate the loss function terms for global and local contrastive learning.

[0034] 5.4) Combining the loss function terms from global and local contrastive learning in step 5.3) and the cross-entropy loss term with word granularity diversity weights in step 5.2), the total loss is obtained, and the parameters are updated based on the total loss:

[0035]

[0036] Where β represents the learning rate, and θ represents all trainable parameters of the visual encoder and cross-modal decoder. This indicates the total loss.

[0037] Furthermore, the calculation of the loss function term for global contrastive learning includes:

[0038] For a specific expresser k, the expression vector on the specific word obtained in step 4.1) is used. Calculate the probability distribution of the word, and then obtain the global contrastive learning loss function term. The calculation formula is as follows:

[0039]

[0040]

[0041] Where softmax(·) represents the activation function, T cThis indicates the number of words decoded before time step t, where τ represents the temperature coefficient. Let N represent the probability distribution of video sentence pairs with speaker k on word c. o It is the meta-training set Zhongyu D i Heyuan Test Set Zhongyu D j Composed of (D) i D j The number of pairs, Denotes the domain D in the o-th pair of domain groups. i The probability distribution over word c H(·|·) represents the loss function term for global contrastive learning, and H(·|·) represents the relative entropy.

[0042] Furthermore, the calculation of the loss function term for local contrastive learning includes:

[0043] The word-granular attention vector obtained by processing all data from the source domain in step 4.1) By combining pairs of samples, several sample pairs (x) are obtained. b ,y b Let A be a set of numbers, and let A be a set of numbers. The formula for calculating the loss function term for local contrastive learning is as follows:

[0044]

[0045] Where ρ(·) represents a distance function, (x b ,y b ) represents the b-th sample pair in set A, and N b It is the number of sample pairs, Y = 1 – [x l =y l ] indicates when x l =y l At that time, [x l =y l [x] = 1, Y = 0; otherwise [x] = 1. l =y l ] = 0, Y = 1; x l and y l x in the sample pair b and y b Each has its own label, and ξ represents a coefficient that controls the magnitude of the distance between the two samples. This represents the loss function term for local contrastive learning.

[0046] Secondly, this invention proposes a visual language translation system based on contrastive learning and word granularity weighting, comprising:

[0047] A lip-reading or finger-reading video preprocessing module is used to extract lip-reading or finger-reading video embedding features from the source domain;

[0048] The natural language text preprocessing module is used to obtain natural language text embedding features;

[0049] The visual encoder module is used to encode the embedded features of lip reading or finger reading videos to obtain the encoded visual features;

[0050] The cross-modal decoder module is used to decode and generate word probability distributions by interacting with the natural language text embedding features through a multi-head attention mechanism on the encoded visual features; in the actual translation stage, the target natural language text is generated autoregressively based on the encoded visual embedding feature vectors.

[0051] The pre-training module is used for the task-based cross-entropy loss function term to perform initial training on the visual encoder and cross-modal decoder;

[0052] The word granularity weight calculation module is used to calculate the word granularity diversity weight of words based on the decoded attention vectors obtained from the initially trained visual encoder and cross-modal decoder.

[0053] The contrast-restricted meta-learning training module is used to randomly partition the source domain into a meta-training set and a meta-test set. In the meta-training phase, the parameters of the visual encoder and cross-modal decoder are updated using a cross-entropy loss term with word granularity diversity weights. In the meta-testing phase, the global and local contrastive learning loss function terms are obtained using the attention vector calculated in the cross-modal decoder after the meta-testing phase. Combined with the cross-entropy loss term with word granularity diversity weights, the parameters of the visual encoder and cross-modal decoder are updated again to obtain the trained visual encoder and cross-modal decoder.

[0054] The translation module is used to acquire the lip-reading or finger-reading video embedding features to be translated in the target domain, and uses a trained visual encoder and cross-modal decoder to complete the visual language translation task of an unseen speaker.

[0055] Compared with the prior art, the beneficial effects of this invention are:

[0056] This invention is a visual language translation method based on contrastive learning and word granular weights. In its implementation, this invention uses a diversity attention mechanism guided by word granular weights and a meta-learning training strategy with contrast constraints.

[0057] (1) By proposing a meta-learning training framework with word granularity diversity weights and contrast constraints, this invention improves the generalization ability of the temporally aligned visual translation system on out-of-domain speakers, clarifies the generalization learning direction of the model in sequence prediction, thereby overcoming the inherent ambiguity between words and adapting to the individual expression habits of speakers, and achieving efficient visual language translation.

[0058] (2) For words with inherent ambiguity that are difficult to identify, this invention proposes a word granularity weight calculation module. It uses the attention vector after the decoder interaction to calculate the word granularity diversity weight to reflect the learning difficulty of words. Then, the model is guided to focus on more difficult words through this difficulty coefficient.

[0059] (3) In the contrast-restricted meta-learning training stage, the global contrast loss function calculated by using the personalized feature vectors of the expresser on specific words maintains the semantic relationship between classes. At the same time, the independence between expressers is improved by using the local contrast loss function, which enhances the generalization ability of the model on unseen expressers, thereby achieving efficient temporally aligned visual language translation.

[0060] In summary, by using a meta-learning training strategy that employs word granularity diversity weights and two complementary contrast constraints, this invention can eliminate inherent word ambiguity while maintaining inter-class relationships in lip-reading and finger-reading translation, and improve the model's generalization ability. It also achieves high-accuracy translation performance even on unseen expressors, thus realizing efficient temporally aligned visual language translation. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the overall framework of the lip-reading or finger-reading translation method of the present invention, wherein... <bos>Indicates the start symbol;

[0062] Figure 2 This is a schematic diagram of the specific process of the word granularity weight calculation module of the present invention;

[0063] Figure 3 This is a schematic diagram illustrating the specific process of the meta-training stage in the meta-learning training strategy of this invention. Detailed Implementation

[0064] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. For ease of explanation, both the encoder and decoder are simplified to single layers; a complete encoder and decoder consists of, for example, [details omitted]. Figure 1 It is composed of multiple stacked layers as shown.

[0065] This invention provides a method for visual language translation tasks based on contrastive learning and word granularity weighting, comprising the following steps:

[0066] Step 1: Extract lip-reading or finger-reading video embedding features and obtain natural language text embedding features;

[0067] Step 2: Divide the complete lip-reading or finger-reading data into source and target domains to ensure that the same speaker cannot appear in both the source and target domains at the same time.

[0068] Step 3: Encode the lip-reading or finger-reading video embedding features using a multi-head attention mechanism visual encoder to obtain the encoded visual features;

[0069] Step 4: For the encoded visual features, interact with the natural language text embedding feature vector through a multi-head attention mechanism to decode and generate word probability distributions, and obtain the task-based cross-entropy loss function term for preliminary training;

[0070] Step 5: Calculate the word granularity diversity weights of words based on the decoded attention vectors obtained from the initially trained visual encoder and cross-modal decoder, and update the cross-entropy loss function term in Step 4 accordingly.

[0071] Step 6: Randomly divide the source domain into a meta-training set and a meta-test set. Use the attention vector calculated in the decoder to obtain the loss function terms for global and local contrastive learning, and calculate the cross-entropy loss function term updated in Step 5.

[0072] Step 7: Combine the loss function terms calculated in Step 6 to obtain the final complete loss function. Then, use the trained task model to complete the lip-reading and finger-reading translation task.

[0073] In a specific embodiment of the present invention, the acquisition of lip-reading or finger-reading video embedding features and natural language text embedding features in step 1 above can be implemented as follows:

[0074] For each given sequence of frames The video (lip reading or finger reading video) is composed of s i T represents the i-th frame. s It is the number of frames in the frame sequence; extracting the video feature matrix. d represents the video feature dimension. In this embodiment, a pre-trained self-supervised representation learning method, such as the AV-HuBERT network, is used to extract the video feature matrix of the lip-reading video, and the ResNet50 network is used to extract the video feature matrix of the finger-reading video.

[0075] For each given natural language sentence, its text sequence is denoted as . l j T represents the j-th word or letter. l It is the number of words, and T l ≤T s Obtain text embedding features In this context, the video frame sequence S and the natural language text sequence L are semantically and temporally aligned, and the goal of visual language translation is to translate the natural language sentence L from the given frame sequence S.

[0076] In a specific embodiment of the present invention, the specific method for segmenting complete lip-reading or finger-reading data in step 2 above is as follows:

[0077] For each speaker, i.e., a lip reader or a finger reader, all their paired video and natural language sentence data can be considered as a domain, represented as N k This represents the number of video-sentence pairs in the k-th domain. Therefore, the complete K domains can be represented as... Complete lip-reading or finger-reading data is divided into a source domain and a target domain. The source domain can be represented as... The target domain can be represented as The source and target domains are strictly divided according to the expresser, and it is guaranteed that the source domain... The expressive that appeared in the text will not be in the target domain. If it reappears in the middle, then ensure Based on the above domain division, all pairs of videos and natural language sentences are numbered 1, 2, ... N. sr +N tg Using the source domain as the training set, denoted as T sr ={S m ,L m |m∈[1,N sr ]}, with the target domain as the test set, is represented as T tg ={S m ,L m |m∈[N sr +1,N sr +N tg ]}, where N sr and N tg These are the amounts of data in the training and testing sets, S. m ,L m This represents the m-th video-sentence pair.

[0078] In one specific embodiment of the present invention, the encoder has a multi-layer structure, and the process of the visual encoder composed of a multi-head attention mechanism described in step 3 above is as follows: Figure 1 As shown, one layer of the encoder consists of a self-attention layer, a feedforward neural network, residual connections, and layer normalization operations stacked together. The implementation process is as follows:

[0079] 3.1) The multi-head attention mechanism is obtained by combining single-head attention mechanisms. When the single-head attention mechanism is computed, it first calculates the single-head attention matrix h based on the input query matrix, original key matrix, original value matrix and their corresponding mapping matrix. i By using different mapping matrix parameters, h single-head attention matrices can be obtained. i The multi-head attention mechanism is obtained by concatenating these parameters and then calculating them with a learnable parameter matrix, according to the following formula:

[0080] In this embodiment, the video feature matrix F obtained in step 1 is used as the query matrix, the original key matrix, and the original value matrix. The calculation of the multi-head attention mechanism is represented as follows:

[0081] MHA(F,F,F)=Concat(h1,h2,…,h h W1

[0082] Where MHA(·) represents the multi-head attention mechanism computation function, and Concat(·) represents the concatenation operation of the single-head attention matrix. Let d represent the learnable parameter matrix, and d represent the video feature dimension.

[0083] 3.2) Construct a self-attention layer based on the multi-head attention mechanism, as shown below:

[0084] SA(F)=MHA(F,F,F)

[0085] In this context, SA(·) represents the self-attention layer.

[0086] 3.3) Using the self-attention layer SA(·) described in step 3.2) and the video feature matrix F obtained in step 1, after multiple residual connections, layer normalization operations, and feedforward neural networks, the encoded video feature matrix F′ is obtained. The specific calculation process is as follows:

[0087] X = LN(F + SA(F))

[0088] FFN(X)=W3σ(W2X)

[0089] F′=LN(X+FFN(X))

[0090] Where LN(·) represents the layer normalization operation, X represents the intermediate result obtained within the encoder layer, FFN(·) represents the feedforward neural network, and σ(·) represents the nonlinear activation function. and They are all learnable parameter matrices. This represents the obtained encoded video feature matrix.

[0091] In one specific embodiment of the present invention, the decoder has a multi-layer structure, and one layer of the decoder is composed of a self-attention layer, a mutual attention layer, a feedforward neural network, residual connections, and layer normalization operations stacked together. The implementation process is as follows:

[0092] 4.1) For the text embedding feature E obtained in step 1, the text embedding feature before time step t will be denoted as... T t This is the number of words decoded before time step t. These words are then processed through the self-attention layer constructed in step 3.4) to update the text embedding features. The calculation process is as follows:

[0093] E′ t =LN(E t +SA(E t ))

[0094] Where SA(·) represents the self-attention layer, and LN(·) represents the layer normalization operation. E′ represents the text embedding feature. t This represents the text embedding features updated via the self-attention layer.

[0095] 4.2) Take the E′ obtained in step 4.1) t The encoded visual features F′ obtained in step 3 are interacted through a mutual attention layer. In the mutual attention layer, E′ t As a query for the multi-head attention mechanism, F′ serves as the original key and value for the multi-head attention mechanism, and the calculation process is as follows:

[0096] I t =LN(E′) t +SA(E′ t ,F′,F′))

[0097] I' t =LN(I t +FFN(I t ))

[0098] Where FFN(·) represents a feedforward neural network, This represents the intermediate results of the mutual attention layer. This indicates the output after decoding.

[0099] 4.3) Utilize the decoded output I′ t The word probability distribution mentioned in step 4 can be calculated using the following formula:

[0100] p t =softmax(W p I' t +b p )

[0101] Where softmax(·) represents the activation function, W p and b p Here, are the weight matrix and bias vector, respectively, and pt represents the calculated probability distribution. The task-based cross-entropy loss function term can be obtained from the calculated probability distribution using the following formula:

[0102]

[0103] Among them, l t The word indicating time step t, l ＜t Words that appear before time step t, l t |l ＜t F represents the condition l ＜t And F generates l t ∑ is the summation symbol. This represents the obtained cross-entropy loss function term.

[0104] During the initial training process, the training set T obtained in step 2 is divided... sr The lip-reading or finger-reading videos and paired natural language sentences are input into the model, and the same operation is performed on each pair of data. Specifically, the video frame sequence S and the natural language sentence L are processed in step 1 to obtain their respective embedding feature matrices F and E; the video feature F is encoded according to step 3 to obtain the encoded video feature F′; F′ and the word sequence embedding feature E are interacted to decode and obtain the word distribution probability p. t This leads to the task-based cross-entropy loss function term. By minimizing this loss function term, the encoder in step 3 and the decoder in step 4 are trained using the gradient descent learning method, preparing for subsequent computations.

[0105] In a specific embodiment of the present invention, the word granularity diversity weight in step 5 above can be obtained in the following way:

[0106] 5.1) The training set T sr The data is input into the model obtained from the initial training in step 4, and then the multi-head attention mechanism MHA(E′) in the last layer of step 4.2) is applied. t The calculation result of F′, F′ is denoted as in T t It is the number of words before time step t. This represents the attention vector of the k-th expresser.

[0107] According to vector u r The personalized expression effect of the kth expressor on word c can be calculated using the following formula:

[0108]

[0109] in, This indicates that when the r-th word is c, It represents the number of samples with the word label 'c'. Let represent the expression vector of the k-th expressor on word c. Based on the expression vectors of different expressors on the same word, their variance is calculated to reflect the degree of ambiguity caused by various behavioral habits. The calculation formula is as follows:

[0110]

[0111] Where K represents the number of expressers, and σ(·) represents a non-linear activation function, such as the sigmoid function. Let represent the diversity weight vector for word 'c', reflecting the learning difficulty the model faces for this word. Therefore, the complete word-granularity diversity weight matrix is ​​obtained by concatenating the weights of each word, represented as follows: T c Indicates the length of the vocabulary.

[0112] 5.2) Apply the word granularity diversity weights obtained in step 5.1) to the cross-entropy loss function term in step 4.3) to obtain the updated cross-entropy loss function term, which is used in the meta-training stage, such as... Figure 2 As shown, the calculation formula is as follows:

[0113]

[0114] Here, * represents a positional multiplication operation between vectors at the word level. It is a cross-entropy loss function with word granularity diversity weighting, which allows the model to adjust its focus to words that are more difficult to learn.

[0115] In one specific embodiment of the present invention, the implementation process of step 6 above is as follows:

[0116] 6.1) The complete source domain containing K domains During each round of training, the dataset is randomly divided into a meta-training set. Heyuan Test Set N tr This indicates the amount of data in the meta-training set. N te This indicates the amount of data in the meta-test set.

[0117] 6.2) As Figure 3 As shown, during the training process using the meta-learning strategy, the meta-training phase minimizes the cross-entropy loss function term obtained in 5.2). The gradient descent learning method is used to update the model parameters. The calculation process is as follows:

[0118]

[0119] Where θ represents all trainable parameters in the model, α represents the learning rate during the meta-training phase, and θ′ represents the updated parameters in the model. The gradient is represented. After the meta-training phase, the performance of expressers on ambiguous words and the semantic space consistency of words are maintained.

[0120] 6.3) During the meta-testing phase, the meta-test set... The data in the model is updated to θ′ by the model parameters. The attention vector calculated in the decoder is used to obtain the loss function terms of global and local contrastive learning mentioned in step 6. The two are complementary in function.

[0121] The purpose of the loss function in global contrastive learning is to solidify the inter-class relationships of words, ensuring that the learned inter-word relationships are preserved on unseen expressors. For a specific expressor k, the expression vector on the specific word obtained in step 5.1) is used. Calculate the probability distribution of the word, and then obtain the global contrastive learning loss function term. The calculation process is as follows:

[0122]

[0123]

[0124] Where softmax(·) represents the activation function, T c This indicates the number of words decoded before time step t, where τ represents the temperature coefficient. Let N represent the probability distribution of video sentence pairs with speaker k on word c. o It is the meta-training set Zhongyu D i Heyuan Test Set Zhongyu D j Composed of (D) i D j The number of pairs, Denotes the domain D in the o-th pair of domain groups. i The probability distribution over word c Similarly, H represents the loss function term for global contrastive learning. H(·|·) represents the relative entropy.

[0125] The purpose of the loss function in local contrastive learning is to mitigate the impact of ambiguous words on the model without considering the expresser, making the model's predictions less sensitive to unseen expressers. This involves the source domain... All sample data are processed through step 5.1) to obtain the word-granularity attention vector. By combining pairs of samples, several sample pairs (x) are obtained. b y b This forms set A. Next, the loss function term for local contrastive learning can be obtained using the following formula:

[0126]

[0127] Where ρ(·) represents a distance function, (x b y b ) represents the b-th sample pair in set A, and N b It is the number of sample pairs, Y = 1 - [x l =y l ] indicates when x l =y l At that time, [x l =y l [x] = 1, Y = 0; otherwise [x] = 1. l =y l ] = 0, Y = 1; x l and y l x in the sample pair b and y b Each has its own label, and ξ represents a coefficient that controls the magnitude of the distance between the two samples. This represents the loss function term for local contrastive learning. In this embodiment, considering computational complexity, all samples are stored in a sorted queue during sampling, and two samples are dequeued each time as a sample pair, instead of enumeration.

[0128] In one specific embodiment of the present invention, the implementation process of step 7 above is as follows:

[0129] The updated cross-entropy loss function obtained in step 5 is combined with the global and local contrastive learning loss function obtained in step 6 to obtain the final complete loss function after combination as described in step 7. The calculation formula is as follows:

[0130]

[0131] Where λ is used to control the balance between the task-based loss function term and the contrastive learning-based loss function term. Let represent the final complete loss function. By minimizing the complete loss function, the initial parameters θ of the model are updated using the gradient descent learning method. The calculation process is as follows:

[0132]

[0133] Here, β represents the learning rate. After parameter updates, a model with better generalization ability is obtained.

[0134] In the target domain In the prediction stage, lip-reading or finger-reading videos of unseen speakers are input into the trained model. The encoder and decoder then predict natural language sentences, thus completing the temporally aligned visual language translation task. This stage does not require the participation of meta-learning.

[0135] The above method will be applied to the following embodiments to demonstrate the technical effects of the present invention. The specific steps in the embodiments will not be repeated.

[0136] This invention conducts experiments on two temporally aligned visual translation tasks: the GRID lip-reading dataset and the ChicagoFSWild finger-reading dataset. The GRID dataset contains 33,000 video sentence pairs recorded by 33 speakers, with a vocabulary consisting of 51 distinct words across 6 categories. To validate the model's performance on unseen speakers, speakers (s1, s2, s20, s22) were selected as the test set. To verify the model's robustness, three additional similar partitions are provided, with unseen speakers in each partition being (s3, s4, s23, s24), (s5, s6, s25, s26), and (s7, s8, s27, s28), respectively. ChicagoFSWild contains 7304 finger-gesturing videos and their corresponding natural language texts from 160 finger-gesturing speakers. All data is divided into three non-overlapping sets of finger-gesturing speakers: 5455 data points from 87 finger-gesturing speakers as the training set, 981 data points from 37 finger-gesturing speakers as the validation set, and 868 data points from 36 finger-gesturing speakers as the test set. The vocabulary size is 31, consisting of 26 English letters and 5 special characters.

[0137] To objectively evaluate the performance of this invention, two evaluation metrics, character error rate (CER) and word error rate (WER), were used in lip-reading translation tasks to assess its effectiveness. The error rate can be calculated using the formula... The calculation yields a value where S, D, and I represent the number of replacement, deletion, and insertion operations when aligning two sequences, respectively, and M is the number of characters or words in the sequence. This invention uses a letter accuracy evaluation metric in the finger-reading translation task, obtained through the following method: S, D, I, M are defined as the same as the lip-reading translation task.

[0138] The implementation details of this invention on the dataset selected for the lip-reading translation task are as follows: The lip-reading video is first processed by a Dlip detector to extract a 100×60 pixel frame sequence centered on the lips as the input video. The dataset is augmented by horizontal flipping with a 50% probability. The implementation details of this invention on the dataset selected for the finger-reading translation task are as follows: The finger-reading video is processed by a face detector to extract a frame sequence centered on the face, and the image size is adjusted to 112×112 as the input data.

[0139] This invention compares with existing models in the lip-reading translation task:

[0140] Compared to the SimulLR model, it achieves state-of-the-art performance in CER and WER metrics on the GRID dataset's conventional partitioning, employing an attention-guided adaptive memory module to achieve synchronous translation.

[0141] In contrast to the LipNet model, this was an earlier model that implemented sentence-level prediction using the CTC method, and it was also the first to attempt speaker non-overlapping segmentation on the GRID dataset.

[0142] Compared to the 3.LCANet model, this is an end-to-end model with cascaded attention and CTC decoder, which partially eliminates the effects of conditional dependencies and accelerates model convergence.

[0143] Compared to the 4.TM-seq2seq model, this model can process videos with or without audio simultaneously, and autoregressively generate target natural language text.

[0144] Compared to the AV-HuBERT model, this model achieves audio and video lip reading recognition by using self-supervised representation learning and masked multimodal clustering.

[0145] Following the steps described in the specific implementation, the experimental results obtained are shown in Table 1. The model of the present invention is represented as CtoML(base) and CtoML, where CtoML(base) represents a simplified model that does not use modules aimed at generalization, i.e., a model trained only with a task-based cross-entropy loss function.

[0146] Table 1: Test results of the lip-reading translation task obtained by this invention on four partitions of the GRID dataset:

[0147]

[0148] Table 1 shows that, across all four partitions, the CtoML model of this invention significantly outperforms state-of-the-art lip-reading models such as SimulLR, LipNet, TM-seq2seq, LCANet, and AV-HuBERT. Specifically, compared to AV-HuBERT, it surpasses AV-HuBERT by an average of 1.44% on the WER (Word Rank Estimation) metric and by an average of 1.36% on the CER (Conversion Rate Estimation) metric across all four partitions. This comparison demonstrates that the word granularity diversity weights provided by this invention facilitate the model's acquisition of learning directions for inherently ambiguous words. Furthermore, the performance of CtoML(base) demonstrates an improvement in domain generalization ability, which was not previously considered by previous models. The performance of the CtoML model is also more stable across different partitions.

[0149] This invention compares a finger-language translation task with the following prior art models:

[0150] Compared to the HDC-FSR model in section 1, this section describes how finger language translation is achieved by training a hand detector and an attention-based encoder and decoder on the ChicagoFSWild dataset.

[0151] Compared to model 2.IAF-FSR, this model translates finger gesture video into natural language text end-to-end, using an iterative attention mechanism that acts on vision to gradually reduce the area of ​​finger gesture actions in the frame sequence.

[0152] In contrast to model 3.FGVA, this is a method that uses a fine-grained attention mechanism to predict natural language text sequentially using a multi-head attention mechanism, and is trained using CTC and cross-entropy loss functions.

[0153] In contrast to model 4.TDC-SL, this model utilizes temporal variable convolutional sequence learning to fuse temporal and spatial features, achieving end-to-end finger-reading translation.

[0154] The abbreviations for the comparative models mentioned above are based on their modeling methods and specific naming conventions, as the original method did not provide abbreviations for these models. Following the steps described in the specific implementation, the experimental results are shown in Table 2.

[0155] Table 2: Test results of the finger translation task obtained by this invention on the ChicagoFSWild dataset on the validation and test sets:

[0156]

[0157] As shown in Table 2, on the test set, CtoML outperforms the previous state-of-the-art model TDC-SL in the finger-reading translation task, with letter accuracy increasing from 50.0% to 54.9%. On the validation set, CtoML's letter accuracy increased from 47.0% in FGVA to 55.7%. This is attributed to the complementary contrastive learning-based constraints in this invention, which enable the model to maintain semantic consistency in the space when the finger-reader is not present while also paying attention to ambiguous words.

[0158] This embodiment also provides a visual language translation system based on contrastive learning and word granularity weights, which is used to implement the above embodiments. The terms "module" and "unit" used below can refer to a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible.

[0159] In this embodiment, the visual language translation system includes:

[0160] A lip-reading or finger-reading video preprocessing module is used to extract lip-reading or finger-reading video embedding features from the source domain;

[0161] The natural language text preprocessing module is used to obtain natural language text embedding features;

[0162] The visual encoder module is used to encode the embedded features of lip reading or finger reading videos to obtain the encoded visual features;

[0163] The cross-modal decoder module is used to decode and generate word probability distributions by interacting with the natural language text embedding features through a multi-head attention mechanism on the encoded visual features; in the actual translation stage, the target natural language text is generated autoregressively based on the encoded visual embedding feature vectors.

[0164] The pre-training module is used for the task-based cross-entropy loss function term to perform initial training on the visual encoder and cross-modal decoder;

[0165] The word granularity weight calculation module is used to calculate the word granularity diversity weight of words based on the decoded attention vectors obtained from the initially trained visual encoder and cross-modal decoder.

[0166] The contrast-restricted meta-learning training module is used to randomly partition the source domain into a meta-training set and a meta-test set. In the meta-training phase, the parameters of the visual encoder and cross-modal decoder are updated using a cross-entropy loss term with word granularity diversity weights. In the meta-testing phase, the global and local contrastive learning loss function terms are obtained using the attention vector calculated in the cross-modal decoder after the meta-testing phase. Combined with the cross-entropy loss term with word granularity diversity weights, the parameters of the visual encoder and cross-modal decoder are updated again to obtain the trained visual encoder and cross-modal decoder.

[0167] The translation module is used to acquire the lip-reading or finger-reading video embedding features to be translated in the target domain, and uses a trained visual encoder and cross-modal decoder to complete the visual language translation task of an unseen speaker.

[0168] For the system embodiments, since they basically correspond to the method embodiments, relevant details are provided in the description of the method embodiments section, and the specific implementation methods of the modules are not repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0169] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0170] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.< / bos>

Claims

1. A visual language translation method based on contrastive learning and word granularity weighting, characterized in that, Includes the following steps: 1) Extract lip-reading or finger-reading video embedding features from the source domain and obtain natural language text embedding features; 2) Encode the lip-reading or finger-reading video embedding features using a visual encoder based on a multi-head attention mechanism to obtain the encoded visual features; 3) For the encoded visual features, the multi-head attention mechanism is used in the cross-modal decoder to interact with the natural language text embedding features, decode and generate word probability distributions, and obtain the task-based cross-entropy loss function term to perform preliminary training on the visual encoder and cross-modal decoder. 4) Based on the decoded attention vectors obtained from the initially trained visual encoder and cross-modal decoder, calculate the word granularity diversity weights and update the cross-entropy loss function term in step 3) accordingly to obtain the cross-entropy loss term with the effect of word granularity diversity weights. Step 4) includes: 4.1) Input the embedded features of the source domain data into the visual encoder and cross-modal decoder initially trained in step 3), and add the mutual attention layer of the last layer of the cross-modal decoder. The calculation result is denoted as ,in, It is a time step The number of words and phrases before Indicates the first Attention vectors of each expresser; The number is calculated using the following formula. An expresser in words Personalized expression effects on: ； in, This indicates that when the r-th word is c, The word tag is The number of samples, Indicates the first An expresser in words The representation vector on; 4.2) Calculate the variance based on the expression vectors of different speakers on the same word. The calculation formula is as follows: ； in, The number of expressers in the source domain. Represents a non-linear activation function. Indicator The diversity weight vector, the complete word-granularity diversity weight matrix is ​​obtained by concatenating the weights of each word, and is represented as: , Indicates the length of the vocabulary; 4.3) Based on the word granularity diversity weight matrix obtained in step 4.2), update the cross-entropy loss function term. The calculation formula is as follows: ； in, The bitwise multiplication operation between vectors at the word granularity. It is a cross-entropy loss function with word granularity diversity weighting; 5) The source domain is randomly divided into a meta-training set and a meta-test set. In the meta-training stage, the parameters of the visual encoder and cross-modal decoder are updated using the cross-entropy loss term with word granularity diversity weights. In the meta-testing stage, the global and local contrastive learning loss function terms are obtained using the attention vector calculated in the cross-modal decoder after the meta-testing stage. Combined with the cross-entropy loss term with word granularity diversity weights, the parameters of the visual encoder and cross-modal decoder are updated again to obtain the trained visual encoder and cross-modal decoder. 6) Obtain the lip-reading or finger-reading video embedding features to be translated in the target domain, and use the trained visual encoder and cross-modal decoder to complete the visual language translation task for the unseen speaker.

2. The visual language translation method based on contrastive learning and word granularity weighting according to claim 1, characterized in that, The lip-reading or finger-reading videos from the source domain are time-aligned with the corresponding natural language text.

3. The visual language translation method based on contrastive learning and word granularity weighting according to claim 1, characterized in that, The visual encoder described is a multi-layer structure, with each layer consisting of a self-attention layer, a feedforward neural network, residual connections, and layer normalization operations stacked together. It is used to encode the embedded features of lip-reading or finger-reading videos and generate an encoded video feature matrix. The first layer calculation process of the visual encoder is represented as follows: ； ； ； in, Standardized operations at the representative level This represents the intermediate results obtained within the encoder layer. Represents a feedforward neural network. Represents a non-linear activation function. and They are all learnable parameter matrices. This represents the obtained encoded video feature matrix. This represents the video embedding features of lip reading or finger reading. This represents the self-attention layer.

4. The visual language translation method based on contrastive learning and word granularity weighting according to claim 1, characterized in that, The cross-modal decoder is a multi-layer structure, with each layer consisting of a self-attention layer, a mutual attention layer, a feedforward neural network, a residual connection, and a layer normalization operation stacked together. It is used to predict words in the next time step based on the natural language text embedding features and the encoded video feature matrix. The first layer computation process of the cross-modal decoder is expressed as follows: ； ； ； ； in, Indicates time step Previous text embedding features, Represents the self-attention layer. Standardized operations at the representative level This represents the text embedding features updated via the self-attention layer. This represents the obtained encoded video feature matrix. This represents the intermediate results of the mutual attention layer. Represents a feedforward neural network. This represents the decoded output. Represents a non-linear activation function. and These are the weight matrix and the bias vector, respectively. This represents the calculated probability distribution.

5. The visual language translation method based on contrastive learning and word granularity weighting according to claim 4, characterized in that, The formula for calculating the task-based cross-entropy loss function is as follows: ； in, Indicates at time step words, Indicates at time step Previous words and phrases Indicates under given conditions and The following generation The probability, This represents the task-based cross-entropy loss function term. Indicates the number of words or phrases.

6. The visual language translation method based on contrastive learning and word granularity weighting according to claim 1, characterized in that, Step 5) includes: 5.1) will contain The source domain data of each domain is randomly divided into a meta-training set during each round of training. Heyuan Test Set , , This indicates the amount of data in the meta-training set. This represents the i-th data point in the meta-training set. , Indicates the amount of data in the meta-test set. This represents the j-th data point in the meta-test set; 5.2) During the meta-training phase, calculate the cross-entropy loss term with the influence of word granularity diversity weights, and update the parameters: ； in, This represents all trainable parameters of the visual encoder and the cross-modal decoder. This represents the learning rate during the meta-training phase. This indicates the updated parameters. Represents the gradient; 5.3) During the meta-testing phase, the data pairs in the meta-test set are updated with model parameters. The model utilizes the attention vector computed in the decoder to calculate the loss function terms for global and local contrastive learning. 5.4) Combining the loss function terms from global and local contrastive learning in step 5.3) and the cross-entropy loss term with word granularity diversity weights in step 5.2), the total loss is obtained, and the parameters are updated based on the total loss: ； in, Indicates the learning rate. This represents all trainable parameters of the visual encoder and the cross-modal decoder. This indicates the total loss.

7. The visual language translation method based on contrastive learning and word granularity weighting according to claim 6, characterized in that, The calculation of the loss function term for global contrastive learning includes: For a specific expresser Using the expression vectors on specific words obtained in step 4.1) Calculate the probability distribution of the word, and then obtain the global contrastive learning loss function term. The calculation formula is as follows: ； ； in, This represents the activation function. Indicates time step The number of words previously decoded Represents the temperature coefficient. Indicates the expresser Video sentences in words probability distribution on, It is the meta-training set Zhongyu Heyuan Test Set Zhongyu Composition The number of correct ones Indicates the first For domains in a domain group In words probability distribution on, The loss function term representing global contrastive learning. Represents relative entropy.

8. The visual language translation method based on contrastive learning and word granularity weighting according to claim 6, characterized in that, The calculation of the loss function term for local contrastive learning includes: The word-granularity attention vector obtained by processing all data from the source domain in step 4.1) By combining pairs of samples, several sample pairs are obtained. , forming a set The formula for calculating the loss function term in local contrastive learning is as follows: ； in, Represent a distance function, Represents a set The Middle One sample pair, It is the number of sample pairs. , indicating when hour, =1, Y=0; otherwise =0, Y=1; and In the sample pair and Each of their own tags A coefficient representing the magnitude of the distance between two samples. This represents the loss function term for local contrastive learning.

9. A visual language translation system based on contrastive learning and word granularity weighting, used to implement the visual language translation method based on contrastive learning and word granularity weighting as described in claim 1, characterized in that the system... include: A lip-reading or finger-reading video preprocessing module is used to extract lip-reading or finger-reading video embedding features from the source domain; The natural language text preprocessing module is used to obtain natural language text embedding features; The visual encoder module is used to encode the embedded features of lip reading or finger reading videos to obtain the encoded visual features; The cross-modal decoder module is used to decode and generate word probability distributions by interacting with the natural language text embedding features through a multi-head attention mechanism on the encoded visual features; in the actual translation stage, the target natural language text is generated autoregressively based on the encoded visual embedding feature vectors. The pre-training module is used for the task-based cross-entropy loss function term to perform initial training on the visual encoder and cross-modal decoder; The word granularity weight calculation module is used to calculate the word granularity diversity weight of words based on the decoded attention vectors obtained from the initially trained visual encoder and cross-modal decoder. The contrast-restricted meta-learning training module is used to randomly partition the source domain into a meta-training set and a meta-test set. In the meta-training phase, the parameters of the visual encoder and cross-modal decoder are updated using a cross-entropy loss term with word granularity diversity weights. In the meta-testing phase, the global and local contrastive learning loss function terms are obtained using the attention vector calculated in the cross-modal decoder after the meta-testing phase. Combined with the cross-entropy loss term with word granularity diversity weights, the parameters of the visual encoder and cross-modal decoder are updated again to obtain the trained visual encoder and cross-modal decoder. The translation module is used to acquire the lip-reading or finger-reading video embedding features to be translated in the target domain, and uses a trained visual encoder and cross-modal decoder to complete the visual language translation task of an unseen speaker.

Images