A multi-granularity sign language video-text retrieval method based on cross-modal alignment

Abstract

A multi-granularity sign language video-text retrieval method based on cross-modal alignment first enhances the sign language video representation through self-supervision. It then optimizes the video segment features by simultaneously considering the correlation between information between and within video segments through temporal attention. It then aggregates the video segment features into coarse-grained sign language video features using temporal covariance pooling. This, combined with fine-grained information, fully exploits the cross-modal, multi-granularity alignment information between the sign language video and text. A shared attention residual Gaussian distribution network is then used to parameterize the sign language video and text into feature distributions. Aligning the feature distributions mitigates the modality gap between the sign language video and text. Finally, fine-grained similarity scores are calculated from the text to the sign language video to retrieve sign language videos that match the query text. This method considers multi-granularity, cross-modal information for the sign language video-text retrieval problem, resulting in highly accurate and effective retrieval results.

Description

Technical Field

[0001] The present invention relates to the field of sign language video retrieval, and in particular to a multi-granularity sign language video-text retrieval method based on cross-modal alignment. Background Art

[0002] Sign language is an important means of communication for the hearing-impaired community, and its understanding and recognition are crucial for promoting information acquisition and communication within this community. Sign language video retrieval focuses on accurately retrieving sign language videos that best match the query text from a closed set, providing hearing-impaired users with a more independent and convenient way to obtain information. As a unique form of language, sign language videos have their own complex grammatical rules. How to achieve precise alignment between sign language videos and text to better serve the deaf community and promote the rich development and technological progress in related fields is the current research content of sign language video-text retrieval. Sign language video-text retrieval technology not only improves the information acquisition ability of the deaf community, but also promotes the integration and development of multimodal learning, artificial intelligence, and barrier-free technology.

[0003] Unlike traditional video-to-text retrieval, which focuses on matching text with video content descriptions, sign language video retrieval focuses on matching natural language with the signer's gestures in the video. Existing sign language video-to-text retrieval methods primarily focus on learning the mapping between sign language video and text in a joint embedding space to capture the interconnectedness between the two modalities. These methods treat sign language retrieval as a cross-lingual retrieval problem and a video-to-text retrieval task, simultaneously considering the linguistic properties of sign language and natural language and identifying fine-grained cross-lingual mappings from signs to words.

[0004] While these methods have made some progress in improving sign language video retrieval performance, they still have limitations in retrieval accuracy. Most methods focus primarily on fine-grained alignment between sign language videos and text, while ignoring the coarse-grained cross-modal relationships between the two. Furthermore, the scarcity of sign language data and the inherent modality differences between sign language videos and text also affect retrieval accuracy. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology and alleviate the problems such as the coarse-grained cross-modal association between sign language videos and texts not being considered in the existing technology, the present invention proposes a multi-granularity sign language video-text retrieval method based on cross-modal alignment. The method uses a temporal attention covariance pooling network to extract coarse-grained sign language video features, and combines them with fine-grained features for modal alignment to capture the multi-granularity similarity between text and video, thereby improving the sign language retrieval performance.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A multi-granularity sign language video-text retrieval method based on cross-modal alignment first enhances the sign language video representation through a self-supervised method; then, through temporal attention, the video clip features are optimized by considering the correlation of information between and within video clips, and temporal covariance pooling is used to aggregate the video clip features into coarse-grained features of the sign language video. The fine-grained information is combined to fully mine the cross-modal multi-granularity alignment information between the sign language video and the text; then, the shared attention residual Gaussian distribution network is used to parameterize the sign language video and text into feature distributions, and the modal gap between the sign language video and the text is alleviated by aligning the feature distributions; finally, the sign language video matching the query text is retrieved by calculating the fine-grained similarity score from the text to the sign language video.

[0008] Furthermore, the method comprises the following steps:

[0009] Step 1: The sign language video set is represented as V = {v1, v2…v i ,…v m}, where m is the number of sign language videos, v1 is the second sign language video, v2 is the second sign language video, and v i is the i-th sign language video, v m For the mth sign language video, the text description set corresponding to the sign language video is represented as T = {t1, t2…t i ,…t m}; query text t i Retrieved sign language video v i The result constitutes a video text pair (t i ,v i );

[0010] Step 2: Select any sign language video v i , two enhanced sign language videos were obtained using random cropping and noise addition methods. and Traverse the entire sign language video collection and use this method to obtain two enhanced sign language video collections respectively and

[0011] Step 3: Use the I3D network as an encoder to extract three sign language video sets {V, V w ,V s The sign language features of each sign language video in

[0012]

[0013] Among them, v i ∈{V,V w ,V s}, h ξrepresents the I3D general domain sign language encoder pre-trained on the BSL-1k dataset, h θ represents the I3D domain-specific sign language encoder used for fine-tuning;

[0014] Step 4: Use CLIP’s image encoder Vision Transformer to extract three sign language video sets {V, V w ,V s The fine-grained features of each sign language video in

[0015]

[0016] Among them, v i ∈{V,V w ,V s}, Indicates sign language video v i The sign language features of the j-th video clip, j = 1, 2..., N v , N v Represents the number of video segments selected at equal intervals in the time dimension in a single video sample;

[0017] Step 5: Calculate the sign language video set {V w ,V s The fine-grained feature similarity matrix of each random pair of sign language videos in}:

[0018]

[0019] in, Indicates the transposition operation, RowSoftmax(·) indicates the softmax calculation is performed row by row;

[0020] Step 6: Calculate the sign language video set {V w ,V s The video-level coarse-grained similarity score for each random pair of sign language videos in}:

[0021]

[0022] Among them, v i ∈V w ,u j ∈V s , Represents the similarity matrix The value of the mth row and nth column in , diag[·] means taking only the diagonal elements in the matrix;

[0023] Step 7. Calculate InfoNCE loss for video similarity learning

[0024]

[0025] Where τ is a learnable temperature parameter and B represents the number of videos in the same batch;

[0026] Step 8: Use CLIP's text encoder Text Transformer to extract fine-grained features of each text description in the text description set T corresponding to the sign language video

[0027]

[0028] Among them, t i ∈T, Represents text t i The kth word, k=1,2...,N t , N t Represents text t i The number of words in the text t, [CLS] indicates the number of words added to the text t i The preceding token; the representation of [CLS] as text t i Coarse-grained features of description

[0029] Step 9: Calculate the intermediate features based on time channel attention for each sign language video in the sign language video set V

[0030]

[0031] Among them, v i ∈V, express The intermediate features of the lth sign language video clip in the time channel attention, express The feature value of the lth sign language video clip in , GAP(·) represents the global average pooling,

[0032] g(·)=Sigmoid(Conv(ReLU(Conv(·))));

[0033] Among them, ReLU(·) represents the first activation function, Sigmoid(·) represents the second activation function, and Conv(·) represents the convolution layer with a convolution kernel size of 1×1; traversal All eigenvalues ​​of

[0034] Step 10: Calculate the intermediate features based on time-space attention for each sign language video in the sign language video set V

[0035]

[0036] Among them, v i∈V, express The intermediate features based on time-space attention of the l-th sign language video clip in , σ represents the softmax operation, BN(·) represents the batch normalization operation, represents element-by-element addition, Represents matrix multiplication; traversal All eigenvalues ​​of

[0037] Step 11: Calculate the sign language features of each sign language video in the sign language video set V after time channel attention and time channel attention calibration of the sign language video clip features.

[0038]

[0039] Among them, v i ∈V, ⊙ represents element-by-element multiplication;

[0040] Step 12: Calculate the coarse-grained features of each sign language video in the sign language video set V

[0041]

[0042] Among them, TCP(·) represents temporal covariance pooling, MPN(·) represents matrix power normalization, and FC(·) represents a fully connected layer;

[0043] Step 13: Calculate the similarity matrix between the fine-grained features of each sign language video in the sign language video set V and the fine-grained features of each text in the text description set T corresponding to the sign language video:

[0044]

[0045] Among them, v i ∈V,t j ∈T;

[0046] Step 14: Calculate the fine-grained text-to-sign language video similarity matrix between each sign language video in the sign language video set V and each text in the text description set T corresponding to the sign language video:

[0047] E t2v (i,j)=E f (i,j)·ColSoftmax(E f (i,j));

[0048] Among them, ColSoftmax(·) indicates that the softmax calculation is performed on a column basis; then the fine-grained similarity score of text to sign language video is expressed as:

[0049] St2v (i,j)=Avg(ColSum(E t2v (i,j)));

[0050] Among them, Avg(·) means finding the average value, and ColSum(·) means summing the columns of the matrix;

[0051] Step 15: Calculate the fine-grained similarity matrix between each sign language video in the sign language video set V and each text in the text description set T:

[0052] E v2t (i,j)=E f (i,j)·RowSoftmax(E f (i,j));

[0053] Then the fine-grained similarity score from sign language video to text is expressed as:

[0054] S v2t (i,j)=Avg(RowSum(E v2t (i,j)));

[0055] Among them, RowSum(·) means the sum of the rows of the matrix;

[0056] Step 16: Calculate the coarse-grained text-to-sign language video similarity score between each sign language video in the sign language video set V and each text in the text description set T:

[0057]

[0058] Among them, |||| represents the modulo operation, Indicates sign language video v i ∈V coarse-grained video features, Represents text t j ∈T coarse-grained sentence features;

[0059] Step 17: Calculate the coarse-grained similarity score between each sign language video in the sign language video set V and each text in the text description set T:

[0060]

[0061] Step 18: Calculate the InfoNCE loss function for fine-grained alignment of text and sign language video

[0062]

[0063] Where τ is the learnable temperature coefficient and B represents the number of video-text pairs in the same batch;

[0064] Step 19: Calculate the InfoNCE loss function for the coarse-grained alignment of text and sign language video.

[0065]

[0066] Step 20: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each sign language video in the sign language video set V:

[0067]

[0068] Among them, v i ∈V, Attn(Q,K,V) is the dot product attention, μ and σ represent The mean vector and variance vector of the feature distribution, F(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution:

[0069] p(z|v)~N(μ,diag(σ 2 ));

[0070] Step 21: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each text in the text description set T:

[0071]

[0072] Among them, t j ∈T, Δμ and Δσ represent Feature distribution relative to The relative position and relative scale of feature distribution, G(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution:

[0073] p(z|v,t)~N(μ+Δμ,diag(σ 2 ·Δσ 2 ));

[0074] Step 22: Calculate the KL divergence loss based on the same noise sampled from standard Gaussian noise and reparameterization,

[0075] L KL =KL(p(z|v,t)||p(z|v));

[0076] Step 23: Calculate the overall loss function.

[0077] L=L f +αL c +βL s +λL KL ;

[0078] Among them, α is the ratio of coarse-grained alignment loss, β is the ratio of video similarity loss, and λ is the ratio of Gaussian distribution alignment loss;

[0079] Step 24: Repeat steps 4 to 23. When L is less than the specified minimum loss value, the calculation ends.

[0080] Step 25: For the query text, select the sign language video with the highest fine-grained similarity score from the text to the sign language video as the retrieval result.

[0081] The technical concept of the present invention is: in sign language video-text retrieval, fully exploit cross-modal multi-granularity alignment information, and use self-supervision methods to enhance sign language video representation, thereby improving the accuracy of sign language video-text retrieval.

[0082] The beneficial effects of the present invention are: it can alleviate the inherent modality differences between sign language videos and texts, enhance the consistency of text information and sign language video information, and improve the accuracy of sign language video-text retrieval tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a flowchart of the multi-granularity sign language video-text retrieval method based on cross-modal alignment. DETAILED DESCRIPTION

[0084] The present invention will be further described below with reference to the accompanying drawings.

[0085] Reference Figure 1 , a multi-granularity sign language video-text retrieval method based on cross-modal alignment, comprising the following steps:

[0086] Step 1: The sign language video set is represented as V = {v1, v2…v i ,…v m}, where m is the number of sign language videos, v1 is the second sign language video, v2 is the second sign language video, and v i is the i-th sign language video, v m For the mth sign language video, the text description set corresponding to the sign language video is represented as T = {t1, t2…t i ,…t m}; query text t i Retrieved sign language video v i The result constitutes a video text pair (t i ,v i );

[0087] Step 2: Select any sign language video v i , two enhanced sign language videos were obtained using random cropping and noise addition methods. and Traverse the entire sign language video collection and use this method to obtain two enhanced sign language video collections respectively and

[0088] Step 3: Use the I3D network as an encoder to extract three sign language video sets {V, V w ,V s The sign language features of each sign language video in

[0089]

[0090] Among them, v i ∈{V,V w ,V s}, h ξ represents the I3D general domain sign language encoder pre-trained on the BSL-1k dataset, h θ represents the I3D domain-specific sign language encoder used for fine-tuning;

[0091] Step 4: Use CLIP’s image encoder Vision Transformer to extract three sign language video sets {V, V w ,V s The fine-grained features of each sign language video in

[0092]

[0093] Among them, v i ∈{V,V w ,V s}, Indicates sign language video v i The sign language features of the j-th video clip, j = 1, 2..., N v , N v Represents the number of video segments selected at equal intervals in the time dimension in a single video sample;

[0094] Step 5: Calculate the sign language video set {V w ,V s The fine-grained feature similarity matrix of each random pair of sign language videos in}:

[0095]

[0096] Among them, v i ∈V w ,u j ∈V s , Indicates the transposition operation, RowSoftmax(·) indicates the softmax calculation is performed row by row;

[0097] Step 6: Calculate the sign language video set {V w ,V s The video-level coarse-grained similarity score for each random pair of sign language videos in}:

[0098]

[0099] Among them, v i ∈V w ,u j ∈V s , Represents the similarity matrix The value of the mth row and nth column in , diag[·] means taking only the diagonal elements in the matrix;

[0100] Step 7. Calculate InfoNCE loss for video similarity learning

[0101]

[0102] Where τ is a learnable temperature parameter and B represents the number of videos in the same batch;

[0103] Step 8: Use CLIP's text encoder Text Transformer to extract fine-grained features of each text description in the text description set T corresponding to the sign language video

[0104]

[0105] Among them, t i ∈T, Represents text t i The kth word, k=1,2...,N t , N t Represents text t i The number of words in the text t, [CLS] indicates the number of words added to the text t i The preceding token; the representation of [CLS] as text t i Coarse-grained features of description

[0106] Step 9: Calculate the intermediate features based on time channel attention for each sign language video in the sign language video set V

[0107]

[0108] Among them, v i ∈V, express The intermediate features of the lth sign language video clip in the time channel attention, express The feature value of the lth sign language video clip in , GAP(·) represents the global average pooling,

[0109] g(·)=Sigmoid(Conv(ReLU(Conv(·))));

[0110] Among them, ReLU(·) represents the first activation function, Sigmoid(·) represents the second activation function, and Conv(·) represents the convolution layer with a convolution kernel size of 1×1; traversal All eigenvalues ​​of

[0111] Step 10: Calculate the intermediate features based on time-space attention for each sign language video in the sign language video set V

[0112]

[0113] Among them, v i ∈V, express The intermediate features based on time-space attention of the l-th sign language video clip in , σ represents the softmax operation, BN(·) represents the batch normalization operation, represents element-by-element addition, Represents matrix multiplication; traversal All eigenvalues ​​of

[0114] Step 11: Calculate the sign language features of each sign language video in the sign language video set V after time channel attention and time channel attention calibration of the sign language video clip features.

[0115]

[0116] Among them, v i ∈V, ⊙ represents element-by-element multiplication;

[0117] Step 12: Calculate the coarse-grained features of each sign language video in the sign language video set V

[0118]

[0119] Among them, TCP(·) represents temporal covariance pooling, MPN(·) represents matrix power normalization, and FC(·) represents a fully connected layer;

[0120] Step 13: Calculate the similarity matrix between the fine-grained features of each sign language video in the sign language video set V and the fine-grained features of each text in the text description set T corresponding to the sign language video:

[0121]

[0122] Among them, v i ∈V,t j ∈T;

[0123] Step 14: Calculate the fine-grained text-to-sign language video similarity matrix between each sign language video in the sign language video set V and each text in the text description set T corresponding to the sign language video:

[0124] E t2v (i,j)=E f (i,j)·ColSoftmax(E f (i,j));

[0125] Among them, ColSoftmax(·) indicates that the softmax calculation is performed on a column basis; then the fine-grained similarity score of text to sign language video is expressed as:

[0126] S t2v (i,j)=Avg(ColSum(E t2v (i,j)));

[0127] Among them, Avg(·) means finding the average value, and ColSum(·) means summing the columns of the matrix;

[0128] Step 15: Calculate the fine-grained similarity matrix between each sign language video in the sign language video set V and each text in the text description set T:

[0129] E v2t (i,j)=E f (i,j)·RowSoftmax(E f (i,j));

[0130] Then the fine-grained similarity score from sign language video to text is expressed as:

[0131] S v2t (i,j)=Avg(RowSum(E v2t (i,j)));

[0132] Among them, RowSum(·) means the sum of the rows of the matrix;

[0133] Step 16: Calculate the coarse-grained text-to-sign language video similarity score between each sign language video in the sign language video set V and each text in the text description set T:

[0134]

[0135] Among them, || || represents the modulo operation, Indicates sign language video v i ∈V coarse-grained video features, Represents text t j ∈T coarse-grained sentence features;

[0136] Step 17: Calculate the coarse-grained similarity score between each sign language video in the sign language video set V and each text in the text description set T:

[0137]

[0138] Step 18: Calculate the InfoNCE loss function for fine-grained alignment of text and sign language video

[0139]

[0140] Where τ is the learnable temperature coefficient and B represents the number of video-text pairs in the same batch;

[0141] Step 19: Calculate the InfoNCE loss function for the coarse-grained alignment of text and sign language video.

[0142]

[0143] Step 20: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each sign language video in the sign language video set V:

[0144]

[0145] Among them, v i ∈V, Attn(Q,K,V) is the dot product attention, μ and σ represent The mean vector and variance vector of the feature distribution, F(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution:

[0146] p(z|v)~N(μ,diag(σ 2 ));

[0147] Step 21: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each text in the text description set T:

[0148]

[0149] Among them, t j ∈T, Δμ and Δσ represent Feature distribution relative to The relative position and relative scale of feature distribution, G(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution:

[0150] p(z|v,t)~N(μ+Δμ,diag(σ 2 ·Δσ 2 ));

[0151] Step 22: Calculate the KL divergence loss based on the same noise sampled from standard Gaussian noise and reparameterization,

[0152] L KL =KL(p(z|v,t)||p(z|v));

[0153] Step 23: Calculate the overall loss function.

[0154] L=L f +αL c +βL s +λL KL ;

[0155] Among them, α is the ratio of coarse-grained alignment loss, β is the ratio of video similarity loss, and λ is the ratio of Gaussian distribution alignment loss;

[0156] Step 24: Repeat steps 4 to 23. When L is less than the specified minimum loss value, the calculation ends.

[0157] Step 25: For the query text, select the sign language video with the highest fine-grained similarity score from the text to the sign language video as the retrieval result.

[0158] In this embodiment, for a sign language video set, a data augmentation operation is first performed to generate an enhanced sign language video set. Then, the sign language encoder and the visual encoder are used to extract the fine-grained features of the sign language video, and the coarse-grained features of the sign language video are obtained through temporal attention. The similarity learning loss L between the enhanced sign language videos is calculated through contrastive learning. s For the query text "I wish you a nice evening.", the text is segmented to obtain the word set {[CLS], i, wish, you, a, nice, evening}. The text encoder is used to extract fine-grained word features and coarse-grained sentence features in the text. The coarse-grained and fine-grained similarity scores between the text and the sign language video are calculated, and the InfoNCE loss function is used to calculate the multi-granularity alignment loss L between the text and the sign language video. f With L c , then the text and sign language video features are parameterized into Gaussian distributions, and the KL divergence loss L is calculated KLThe feature distributions of the two are aligned, and the fine-grained similarity scores of the query text and the sign language video are used to select the video with the highest score from the sign language video collection. The retrieved result is the sign language video that correctly corresponds to the query text.

[0159] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A multi-granularity sign language video-text retrieval method based on cross-modal alignment, characterized by: First, the self-supervision method is used to enhance the representation of sign language videos. Then, the temporal attention is used to simultaneously consider the correlation of information between and within video clips to optimize the video clip features. The temporal covariance pooling is used to aggregate the video clip features into coarse-grained features of the sign language video. The fine-grained information is combined to fully mine the cross-modal multi-granularity alignment information between the sign language video and the text. The image encoder Vision Transformer of CLIP is used to extract the fine-grained features of each sign language video in the sign language video collection. The text encoder Text Transformer extracts the fine-grained features of each text description in the set of text descriptions corresponding to the sign language video, and calculates the similarity matrix between the fine-grained features of each sign language video in the sign language video set and the fine-grained features of each text in the set of text descriptions corresponding to the sign language video; then, the shared attention residual Gaussian distribution network is used to parameterize the sign language video and text into feature distributions, and the modal gap between the sign language video and the text is alleviated by aligning the feature distributions. The loss function for the fine-grained alignment of text and sign language video and the loss function for the coarse-grained alignment of text and sign language video are calculated; finally, the fine-grained similarity score from text to sign language video is calculated to retrieve sign language videos that match the query text.

2. The multi-granularity sign language video-text retrieval method based on cross-modal alignment according to claim 1, characterized in that: The method comprises the following steps: Step 1: The sign language video collection is represented as ,in m is the number of sign language videos, This is the first sign language video. For the second sign language video, is the i-th sign language video, For the m sign language videos, and the text description set corresponding to the sign language video is represented as ; Query text Retrieved sign language videos The results constitute video-text pairs ; Step 2: Select any sign language video , two enhanced sign language videos were obtained using random cropping and noise addition methods. and ; Traverse the entire sign language video collection and use this method to obtain two enhanced sign language video collections respectively and ; Step 3: Use the I3D network as an encoder to extract three sign language video sets Sign language features of each sign language video ； in, , represents the I3D general domain sign language encoder pre-trained on the BSL-1k dataset, represents the I3D domain-specific sign language encoder used for fine-tuning; Step 4: Use CLIP’s image encoder Vision Transformer to extract three sign language video sets The fine-grained features of each sign language video in ； in, , Sign language video The sign language features of the j-th video clip, j=1,2..., , Represents the number of video segments selected at equal intervals in the time dimension in a single video sample; Step 5: Calculate the sign language video collection The fine-grained feature similarity matrix of each random pair of sign language videos in : ； in, , , represents the transpose operation, Indicates that softmax calculation is performed row by row; Step 6. Calculate the sign language video collection The video-level coarse-grained similarity score for each random pair of sign language videos in : ； in, , , Represents the similarity matrix The value of row m and column n in Indicates that only the diagonal elements in the matrix are taken; Step 7. Calculate InfoNCE loss for video similarity learning ； in, is a learnable temperature parameter, and B represents the number of videos in the same batch; Step 8. Use CLIP's text encoder Text Transformer to extract the text description set corresponding to the sign language video The fine-grained features of each text description in ； in, , Represents text The kth word, k=1,2..., , Represents text The number of words in the text, [CLS] indicates the number of words added to the text. Previous token; [CLS] representation as text Coarse-grained features of description ; Step 9. Calculate the sign language video collection The intermediate features of each sign language video in the time channel attention ； in, , express Middle Intermediate features based on temporal channel attention of sign language video clips, express Middle The feature value of the sign language video clip, represents global average pooling, ； in, represents the first activation function, represents the second activation function, Represents a convolution layer with a convolution kernel size of 1×1; traverse All eigenvalues ​​of ; Step 10: Calculate the sign language video collection The intermediate features of each sign language video in the time-space attention ； in, , express Middle The intermediate features of the sign language video clips based on temporal and spatial attention, represents the softmax operation, represents the batch normalization operation, represents element-by-element addition, Represents matrix multiplication; traversal All eigenvalues ​​of ; Step 11: Calculate the sign language video collection The sign language features of each sign language video in the time channel attention and the sign language video clip features calibrated by the time channel attention ； in, , ⊙ represents element-by-element multiplication; Step 12: Calculate the sign language video collection The coarse-grained features of each sign language video in ； in, represents temporal covariance pooling, represents matrix power normalization, represents the fully connected layer; Step 13: Calculate the sign language video collection The fine-grained features of each sign language video and the corresponding text description set of the sign language video The similarity matrix of the fine-grained features of each text in: ； in, , ; Step 14: Calculate the sign language video collection Each sign language video and the text description set corresponding to the sign language video The fine-grained similarity matrix of text to sign language video between each text in: ； in, Indicates that softmax calculation is performed column by column; then the fine-grained similarity score of text to sign language video is expressed as: ； in, Indicates the average value, Indicates the sum of the matrix columns; Step 15: Calculate the sign language video collection Each sign language video and text description collection in The fine-grained similarity matrix of sign language video to text between each text in: ； Then the fine-grained similarity score from sign language video to text is expressed as: ； in, Indicates the sum of the matrix rows; Step 16: Calculate the sign language video collection Each sign language video and text description collection in The coarse-grained similarity score of the text-to-sign language video between each text in : ； in, Represents the modulo operation, Sign language video Coarse-grained video features, Represents text Coarse-grained sentence features; Step 17: Calculate the sign language video collection Each sign language video and text description collection in The coarse-grained similarity score of the sign language video to text between each text in : ； Step 18: Calculate the InfoNCE loss function for fine-grained alignment of text and sign language video ； in, is the learnable temperature coefficient, B represents the number of video-text pairs in the same batch; Step 19: Calculate the InfoNCE loss function for the coarse-grained alignment of text and sign language video. ； Step 20: Use the Shared Attention Residual Gaussian Distribution Network to obtain a collection of sign language videos The mean and variance of the fine-grained feature distribution of each sign language video in: ； in, , is the dot-product attention, and Respectively The mean and variance vectors of the feature distribution, represents a linear network layer; then Parameterized as a multivariate Gaussian distribution: ； Step 21: Use the shared attention residual Gaussian distribution network to obtain a text description set The mean and variance of the fine-grained feature distribution of each text in: ； in, , and express Feature distribution relative to The relative position and relative scale of the feature distribution, represents a linear network layer; then Parameterized as a multivariate Gaussian distribution: ； Step 22: Calculate the KL divergence loss based on the same noise sampled from standard Gaussian noise and reparameterization, ； Step 23: Calculate the overall loss function. ； in, is the ratio of the coarse-grained alignment loss, is the ratio of video similarity loss, is the ratio of Gaussian distribution alignment loss; Step 24: Repeat steps 4 to 23. When the loss is less than the specified minimum value, the calculation ends; Step 25: For the query text, select the sign language video with the highest fine-grained similarity score from the text to the sign language video as the retrieval result.

Images