Multi-granularity sign language video-text retrieval method based on cross-modal alignment

Abstract

The invention discloses a multi-granularity sign language video-text retrieval method based on cross-modal alignment. Firstly, sign language video representation is enhanced through a self-supervision method; secondly, optimizing video clip features by considering the correlation of information between video clips and information in the video clips through time attention, aggregating the video clip features into sign language video coarse-grained features by utilizing time covariance pooling, and fully mining cross-modal multi-granularity alignment information between sign language videos and texts in combination with fine-grained information; parameterizing the sign language video and the text into feature distribution by using a shared attention residual Gaussian distribution network, and relieving a modal gap between the sign language video and the text by aligning the feature distribution; and finally, searching the sign language video matched with the query text by calculating the fine-grained similarity score from the text to the sign language video. The multi-granularity cross-modal information of the sign language video-text retrieval problem is considered, the retrieval result is high in accuracy, and the effect is good.

Description

Technical Field

[0001] The present invention relates to the field of sign language video retrieval, and in particular to a multi-granularity sign language video-text retrieval method based on cross-modal alignment. Background Art

[0002] Sign language is an important way of communication for the hearing-impaired community, and its understanding and recognition are crucial to promoting information acquisition and communication in this community. Sign language video retrieval focuses on accurately retrieving sign language videos that best match the query text from a closed set, providing hearing-impaired users with a more independent and convenient way to obtain information. As a unique form of language, sign language videos have their own complex grammatical rules. How to achieve precise alignment between sign language videos and texts to better serve the deaf community and promote the rich development and technological progress in related fields is the current research content of sign language video-text retrieval. Sign language video-text retrieval technology not only improves the information acquisition ability of the deaf community, but also promotes the integration and development of multimodal learning, artificial intelligence and barrier-free technology.

[0003] Unlike traditional video text retrieval that focuses on matching text with video content description, sign language video retrieval focuses more on matching natural language with the signer's gestures in the video. Existing sign language video-text retrieval methods mainly focus on learning the mapping relationship between sign language video and text in a joint embedding space to capture the mutual connection between the two modalities. These methods regard sign language retrieval as a cross-language retrieval problem and video text retrieval task, taking into account the linguistic properties of sign language and natural language, and identifying fine-grained cross-language mappings from signs to words.

[0004] Although these methods have made some progress in improving the performance of sign language video retrieval, they still have limitations in retrieval accuracy. Most methods focus on the fine-grained alignment between sign language videos and texts, while ignoring the coarse-grained cross-modal association between the two. In addition, the scarcity of sign language data and the inherent modality differences between sign language videos and texts also affect retrieval accuracy. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art and to alleviate the problem that the coarse-grained cross-modal association between sign language video and text is not considered in the prior art, the present invention proposes a multi-granularity sign language video-text retrieval method based on cross-modal alignment, which uses a temporal attention covariance pooling network to extract coarse-grained sign language video features, and combines fine-grained features for modal alignment to capture the multi-granularity similarities between text and video, thereby improving the sign language retrieval performance.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A multi-granularity sign language video-text retrieval method based on cross-modal alignment first enhances the sign language video representation through a self-supervised method; then optimizes the video clip features by considering the correlation between and within video clips through temporal attention, aggregates the video clip features into coarse-grained features of the sign language video using temporal covariance pooling, and fully mines the cross-modal multi-granularity alignment information between the sign language video and the text by combining fine-grained information; then uses a shared attention residual Gaussian distribution network to parameterize the sign language video and text into feature distributions, and alleviates the modal gap between the sign language video and the text by aligning the feature distributions; finally, retrieves the sign language video matching the query text by calculating the fine-grained similarity score from the text to the sign language video.

[0008] Further, the method comprises the following steps:

[0009] Step 1: The sign language video set is represented as V = {v1,v2…v i ,…v m}, where m is the number of sign language videos, v1 is the second sign language video, v2 is the second sign language video, and v i is the i-th sign language video, v m is the mth sign language video, and the text description set corresponding to the sign language video is represented as T = {t1, t2…t i ,…t m}; Query text t i Retrieved sign language videos v i The result constitutes a video text pair (t i ,v i );

[0010] Step 2: Select any sign language video v i , two enhanced sign language videos were obtained by using random cropping and adding noise respectively and Traverse the entire sign language video collection and use this method to obtain two enhanced sign language video collections respectively. and

[0011] Step 3: Use the I3D network as an encoder to extract three sign language video sets {V, V w ,V s The sign language features of each sign language video in}

[0012]

[0013] Among them, v i ∈{V,V w ,V s}, h ξrepresents the I3D general domain sign language encoder pre-trained on the BSL-1k dataset, h θ represents the I3D domain-specific sign language encoder for fine-tuning;

[0014] Step 4: Use CLIP’s image encoder Vision Transformer to extract three sign language video sets {V, V w ,V s The fine-grained features of each sign language video in

[0015]

[0016] Among them, v i ∈{V,V w ,V s}, Indicates sign language video v i The sign language features of the jth video clip, j = 1, 2..., N v , N v Represents the number of video segments selected at equal intervals in the time dimension in a single video sample;

[0017] Step 5: Calculate the sign language video set {V w ,V s The fine-grained feature similarity matrix of each random pair of sign language videos in}:

[0018]

[0019] in, represents the transposition operation, and RowSoftmax(·) represents the softmax calculation performed row by row;

[0020] Step 6: Calculate the sign language video set {V w ,V s The video-level coarse-grained similarity score for each random pair of sign language videos in}:

[0021]

[0022] Among them, v i ∈V w ,u j ∈V s , Represents the similarity matrix The value of the mth row and nth column in , diag[·] means taking only the diagonal elements in the matrix;

[0023] Step 7: Calculate the InfoNCE loss for video similarity learning

[0024]

[0025] Among them, τ is a learnable temperature parameter, B represents the number of videos in the same batch;

[0026] Step 8: Use CLIP's text encoder Text Transformer to extract fine-grained features of each text description in the text description set T corresponding to the sign language video

[0027]

[0028] Among them, t i ∈T, Represents text t i The kth word of t , N t Represents text t i The number of words in the text t. [CLS] indicates the number of words added to the text t i Previous token; the representation of [CLS] as text t i Coarse-grained features of description

[0029] Step 9: Calculate the intermediate features based on time channel attention for each sign language video in the sign language video set V

[0030]

[0031] Among them, v i ∈V, express The intermediate features based on temporal channel attention of the lth sign language video clip in , express The feature value of the lth sign language video clip in , GAP(·) represents the global average pooling,

[0032] g(·)=Sigmoid(Conv(ReLU(Conv(·))));

[0033] Among them, ReLU(·) represents the first activation function, Sigmoid(·) represents the second activation function, and Conv(·) represents the convolution layer with a convolution kernel size of 1×1; traverse All eigenvalues ​​of

[0034] Step 10: Calculate the intermediate features based on time-space attention for each sign language video in the sign language video set V

[0035]

[0036] Among them, v i∈V, express The intermediate features of the l-th sign language video clip based on time-space attention in , σ represents the softmax operation, BN(·) represents the batch normalization operation, represents element-by-element addition, Represents matrix multiplication; traversal All eigenvalues ​​of

[0037] Step 11: Calculate the sign language features of each sign language video in the sign language video set V. The sign language video clip features after time channel attention and time channel attention calibration

[0038]

[0039] Among them, v i ∈V, ⊙ represents element-by-element multiplication;

[0040] Step 12: Calculate the coarse-grained features of each sign language video in the sign language video set V

[0041]

[0042] Among them, TCP(·) represents temporal covariance pooling, MPN(·) represents matrix power normalization, and FC(·) represents a fully connected layer;

[0043] Step 13: Calculate the similarity matrix between the fine-grained features of each sign language video in the sign language video set V and the fine-grained features of each text in the text description set T corresponding to the sign language video:

[0044]

[0045] Among them, v i ∈V,t j ∈T;

[0046] Step 14: Calculate the fine-grained similarity matrix from text to sign language video between each sign language video in the sign language video set V and each text in the text description set T corresponding to the sign language video:

[0047] E t2v (i,j)=E f (i,j)·ColSoftmax(E f (i,j));

[0048] Among them, ColSoftmax(·) indicates that softmax calculation is performed by column; then the fine-grained similarity score from text to sign language video is expressed as:

[0049] St2v (i,j)=Avg(ColSum(E t2v (i,j)));

[0050] Among them, Avg(·) means finding the average value, and ColSum(·) means summing the columns of the matrix;

[0051] Step 15: Calculate the fine-grained similarity matrix from sign language video to text between each sign language video in the sign language video set V and each text in the text description set T:

[0052] E v2t (i,j)=E f (i,j)·RowSoftmax(E f (i,j));

[0053] Then the fine-grained similarity score from sign language video to text is expressed as:

[0054] S v2t (i,j)=Avg(RowSum(E v2t (i,j)));

[0055] Among them, RowSum(·) means the sum of the rows of the matrix;

[0056] Step 16: Calculate the coarse-grained similarity score between each sign language video in the sign language video set V and each text in the text description set T:

[0057]

[0058] Among them, |||| represents the modulo operation, Indicates sign language video v i ∈V, a coarse-grained video feature. Represents text t j ∈T coarse-grained sentence features;

[0059] Step 17: Calculate the coarse-grained similarity score between each sign language video in the sign language video set V and each text in the text description set T:

[0060]

[0061] Step 18: Calculate the InfoNCE loss function for fine-grained alignment of text and sign language video

[0062]

[0063] Where τ is the learnable temperature coefficient and B represents the number of video-text pairs in the same batch;

[0064] Step 19: Calculate the InfoNCE loss function for the coarse-grained alignment of text and sign language video.

[0065]

[0066] Step 20: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each sign language video in the sign language video set V:

[0067]

[0068] Among them, v i ∈V, Attn(Q,K,V) is the dot product attention, μ and σ represent The mean vector and variance vector of the feature distribution, F(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution:

[0069] p(z|v)~N(μ,diag(σ 2 ));

[0070] Step 21: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each text in the text description set T:

[0071]

[0072] Among them, t j ∈T, Δμ and Δσ represent The distribution of features relative to The relative position and relative scale of feature distribution, G(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution:

[0073] p(z|v,t)~N(μ+Δμ,diag(σ 2 ·Δσ 2 ));

[0074] Step 22: Calculate the KL divergence loss based on the same noise sampled from standard Gaussian noise and reparameterization,

[0075] L KL =KL(p(z|v,t)||p(z|v));

[0076] Step 23: Calculate the overall loss function.

[0077] L=L f +αL c +βL s +λL KL ;

[0078] Among them, α is the ratio of coarse-grained alignment loss, β is the ratio of video similarity loss, and λ is the ratio of Gaussian distribution alignment loss;

[0079] Step 24: Repeat steps 4 to 23. When L is less than the specified minimum loss value, the calculation ends.

[0080] Step 25: For the query text, select the sign language video with the highest fine-grained similarity score from the text to the sign language video as the retrieval result.

[0081] The technical concept of the present invention is: in sign language video-text retrieval, fully exploit cross-modal multi-granularity alignment information, and use self-supervision methods to enhance sign language video representation, thereby improving the accuracy of sign language video-text retrieval.

[0082] The beneficial effects of the present invention are: it can alleviate the inherent modal differences between sign language video and text, enhance the consistency of text information and sign language video information, and improve the accuracy of sign language video-text retrieval tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a flowchart of a multi-granularity sign language video-text retrieval method based on cross-modal alignment. DETAILED DESCRIPTION

[0084] The present invention will be further described below in conjunction with the accompanying drawings.

[0085] Reference Figure 1 , a multi-granularity sign language video-text retrieval method based on cross-modal alignment, comprising the following steps:

[0086] Step 1: The sign language video set is represented as V = {v1,v2…v i ,…v m}, where m is the number of sign language videos, v1 is the second sign language video, v2 is the second sign language video, and v i is the i-th sign language video, v m is the mth sign language video, and the text description set corresponding to the sign language video is represented as T = {t1, t2…t i ,…t m}; Query text t i Retrieved sign language videos v i The result constitutes a video text pair (t i ,v i );

[0087] Step 2: Select any sign language video v i , two enhanced sign language videos were obtained by using random cropping and adding noise respectively and Traverse the entire sign language video collection and use this method to obtain two enhanced sign language video collections respectively. and

[0088] Step 3: Use the I3D network as an encoder to extract three sign language video sets {V, V w ,V s The sign language features of each sign language video in}

[0089]

[0090] Among them, v i ∈{V,V w ,V s}, h ξ represents the I3D general domain sign language encoder pre-trained on the BSL-1k dataset, h θ represents the I3D domain-specific sign language encoder for fine-tuning;

[0091] Step 4: Use CLIP’s image encoder Vision Transformer to extract three sign language video sets {V, V w ,V s The fine-grained features of each sign language video in

[0092]

[0093] Among them, v i ∈{V,V w ,V s}, Indicates sign language video v i The sign language features of the jth video clip, j = 1, 2..., N v , N v Represents the number of video segments selected at equal intervals in the time dimension in a single video sample;

[0094] Step 5: Calculate the sign language video set {V w ,V s The fine-grained feature similarity matrix of each random pair of sign language videos in}:

[0095]

[0096] Among them, v i ∈V w ,u j ∈V s , represents the transposition operation, and RowSoftmax(·) represents the softmax calculation performed row by row;

[0097] Step 6: Calculate the sign language video set {V w ,V s The video-level coarse-grained similarity score for each random pair of sign language videos in}:

[0098]

[0099] Among them, v i ∈V w ,u j ∈V s , Represents the similarity matrix The value of the mth row and nth column in , diag[·] means taking only the diagonal elements in the matrix;

[0100] Step 7: Calculate the InfoNCE loss for video similarity learning

[0101]

[0102] Among them, τ is a learnable temperature parameter, B represents the number of videos in the same batch;

[0103] Step 8: Use CLIP's text encoder Text Transformer to extract fine-grained features of each text description in the text description set T corresponding to the sign language video

[0104]

[0105] Among them, t i ∈T, Represents text t i The kth word of t , N t Represents text t i The number of words in the text t. [CLS] indicates the number of words added to the text t i Previous token; the representation of [CLS] as text t i Coarse-grained features of description

[0106] Step 9: Calculate the intermediate features based on time channel attention for each sign language video in the sign language video set V

[0107]

[0108] Among them, v i ∈V, express The intermediate features based on temporal channel attention of the lth sign language video clip in , express The feature value of the lth sign language video clip in , GAP(·) represents the global average pooling,

[0109] g(·)=Sigmoid(Conv(ReLU(Conv(·))));

[0110] Among them, ReLU(·) represents the first activation function, Sigmoid(·) represents the second activation function, and Conv(·) represents the convolution layer with a convolution kernel size of 1×1; traverse All eigenvalues ​​of

[0111] Step 10: Calculate the intermediate features based on time-space attention for each sign language video in the sign language video set V

[0112]

[0113] Among them, v i ∈V, express The intermediate features of the l-th sign language video clip based on time-space attention in , σ represents the softmax operation, BN(·) represents the batch normalization operation, represents element-by-element addition, Represents matrix multiplication; traversal All eigenvalues ​​of

[0114] Step 11: Calculate the sign language features of each sign language video in the sign language video set V. The sign language video clip features after time channel attention and time channel attention calibration

[0115]

[0116] Among them, v i ∈V, ⊙ represents element-by-element multiplication;

[0117] Step 12: Calculate the coarse-grained features of each sign language video in the sign language video set V

[0118]

[0119] Among them, TCP(·) represents temporal covariance pooling, MPN(·) represents matrix power normalization, and FC(·) represents a fully connected layer;

[0120] Step 13: Calculate the similarity matrix between the fine-grained features of each sign language video in the sign language video set V and the fine-grained features of each text in the text description set T corresponding to the sign language video:

[0121]

[0122] Among them, v i ∈V,t j ∈T;

[0123] Step 14: Calculate the fine-grained similarity matrix from text to sign language video between each sign language video in the sign language video set V and each text in the text description set T corresponding to the sign language video:

[0124] E t2v (i,j)=E f (i,j)·ColSoftmax(E f (i,j));

[0125] Among them, ColSoftmax(·) indicates that softmax calculation is performed by column; then the fine-grained similarity score from text to sign language video is expressed as:

[0126] S t2v (i,j)=Avg(ColSum(E t2v (i,j)));

[0127] Among them, Avg(·) means finding the average value, and ColSum(·) means summing the columns of the matrix;

[0128] Step 15: Calculate the fine-grained similarity matrix from sign language video to text between each sign language video in the sign language video set V and each text in the text description set T:

[0129] E v2t (i,j)=E f (i,j)·RowSoftmax(E f (i,j));

[0130] Then the fine-grained similarity score from sign language video to text is expressed as:

[0131] S v2t (i,j)=Avg(RowSum(E v2t (i,j)));

[0132] Among them, RowSum(·) means the sum of the rows of the matrix;

[0133] Step 16: Calculate the coarse-grained similarity score between each sign language video in the sign language video set V and each text in the text description set T:

[0134]

[0135] Among them, || || represents the modulo operation, Indicates sign language video v i ∈V, a coarse-grained video feature. Represents text t j ∈T coarse-grained sentence features;

[0136] Step 17: Calculate the coarse-grained similarity score between each sign language video in the sign language video set V and each text in the text description set T:

[0137]

[0138] Step 18: Calculate the InfoNCE loss function for fine-grained alignment of text and sign language video

[0139]

[0140] Where τ is the learnable temperature coefficient and B represents the number of video-text pairs in the same batch;

[0141] Step 19: Calculate the InfoNCE loss function for the coarse-grained alignment of text and sign language video.

[0142]

[0143] Step 20: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each sign language video in the sign language video set V:

[0144]

[0145] Among them, v i ∈V, Attn(Q,K,V) is the dot product attention, μ and σ represent The mean vector and variance vector of the feature distribution, F(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution:

[0146] p(z|v)~N(μ,diag(σ 2 ));

[0147] Step 21: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each text in the text description set T:

[0148]

[0149] Among them, t j ∈T, Δμ and Δσ represent The distribution of features relative to The relative position and relative scale of feature distribution, G(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution:

[0150] p(z|v,t)~N(μ+Δμ,diag(σ 2 ·Δσ 2 ));

[0151] Step 22: Calculate the KL divergence loss based on the same noise sampled from standard Gaussian noise and reparameterization,

[0152] L KL =KL(p(z|v,t)||p(z|v));

[0153] Step 23: Calculate the overall loss function.

[0154] L=L f +αL c +βL s +λL KL ;

[0155] Among them, α is the ratio of coarse-grained alignment loss, β is the ratio of video similarity loss, and λ is the ratio of Gaussian distribution alignment loss;

[0156] Step 24: Repeat steps 4 to 23. When L is less than the specified minimum loss value, the calculation ends.

[0157] Step 25: For the query text, select the sign language video with the highest fine-grained similarity score from the text to the sign language video as the retrieval result.

[0158] In this embodiment, for a sign language video set, a data augmentation operation is first performed to generate an enhanced sign language video set, and then the sign language encoder and the visual encoder are used to extract the fine-grained features of the sign language video, and the coarse-grained features of the sign language video are obtained through temporal attention. The similarity learning loss L between the enhanced sign language videos is calculated through contrastive learning. s ; For the query text "I wish you a nice evening.", the text is segmented to obtain the word set {[CLS], i, wish, you, a, nice, evening}, and the text encoder is used to extract the fine-grained word features and coarse-grained sentence features in the text. The coarse-grained and fine-grained similarity scores between the text and the sign language video are calculated, and the InfoNCE loss function is used to calculate the multi-granular alignment loss L between the text and the sign language video. f With L c Then, the text and sign language video features are parameterized into Gaussian distributions, and the KL divergence loss L is calculated. KLThe feature distributions of the two are aligned, and the video with the highest score is selected from the sign language video collection through the fine-grained similarity scores of the query text and the sign language video. The retrieved result is the sign language video that correctly corresponds to the query text.

[0159] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A multi-granularity sign language video-text retrieval method based on cross-modal alignment, characterized in that: First, the representation of sign language video is enhanced through self-supervision method. Then, temporal attention is used to simultaneously consider the correlation of information between and within video clips to optimize the features of video clips. Temporal covariance pooling is used to aggregate the features of video clips into coarse-grained features of sign language video, and fine-grained information is combined to fully mine the cross-modal multi-granularity alignment information between sign language video and text. Then, a shared attention residual Gaussian distribution network is used to parameterize sign language video and text into feature distributions, and the modal gap between sign language video and text is alleviated by aligning feature distributions. Finally, the fine-grained similarity score from text to sign language video is calculated to retrieve sign language videos that match the query text.

2. The multi-granularity sign language video-text retrieval method based on cross-modal alignment as claimed in claim 1, characterized in that: The method comprises the following steps: Step 1: The sign language video set is represented as V = {v1,v2…v i ,…v m }, where m is the number of sign language videos, v1 is the second sign language video, v2 is the second sign language video, and v i is the i-th sign language video, v m is the mth sign language video, and the text description set corresponding to the sign language video is represented as T = {t1, t2…t i ,…t m }; Query text t i Retrieved sign language videos v i The result constitutes a video text pair (t i ,v i ); Step 2: Select any sign language video v i , two enhanced sign language videos were obtained by using random cropping and adding noise respectively and Traverse the entire sign language video collection and use this method to obtain two enhanced sign language video collections respectively. and Step 3: Use the I3D network as an encoder to extract three sign language video sets {V, V w ,V s The sign language features of each sign language video in} Among them, v i ∈{V,V w ,V s }, h ξ represents the I3D general domain sign language encoder pre-trained on the BSL-1k dataset, h θ represents the I3D domain-specific sign language encoder for fine-tuning; Step 4: Use CLIP’s image encoder Vision Transformer to extract three sign language video sets {V, V w ,V s The fine-grained features of each sign language video in Among them, v i ∈{V,V w ,V s }, Indicates sign language video v i The sign language features of the jth video clip, j = 1, 2..., N v , N v Represents the number of video segments selected at equal intervals in the time dimension in a single video sample; Step 5: Calculate the sign language video set {V w ,V s The fine-grained feature similarity matrix of each random pair of sign language videos in}: Among them, v i ∈V w ,u j ∈V s , represents the transposition operation, and RowSoftmax(·) represents the softmax calculation performed row by row; Step 6: Calculate the sign language video set {V w ,V s The video-level coarse-grained similarity score for each random pair of sign language videos in}: Among them, v i ∈V w ,u j ∈V s , Represents the similarity matrix The value of the mth row and nth column in , diag[·] means taking only the diagonal elements in the matrix; Step 7: Calculate the InfoNCE loss for video similarity learning Among them, τ is a learnable temperature parameter, B represents the number of videos in the same batch; Step 8: Use CLIP's text encoder Text Transformer to extract fine-grained features of each text description in the text description set T corresponding to the sign language video Among them, t i ∈T, Represents text t i The kth word of t , N t Represents text t i The number of words in the text t. [CLS] indicates the number of words added to the text t i Previous token; the representation of [CLS] as text t i Coarse-grained features of description Step 9: Calculate the intermediate features based on time channel attention for each sign language video in the sign language video set V Among them, v i ∈V, express The intermediate features based on temporal channel attention of the lth sign language video clip in , express The feature value of the lth sign language video clip in , GAP(·) represents the global average pooling, g(·)=Sigmoid(Conv(ReLU(Conv(·)))); Among them, ReLU(·) represents the first activation function, Sigmoid(·) represents the second activation function, and Conv(·) represents the convolution layer with a convolution kernel size of 1×1; traverse All eigenvalues ​​of Step 10: Calculate the intermediate features based on time-space attention for each sign language video in the sign language video set V Among them, v i ∈V, express The intermediate features of the l-th sign language video clip based on time-space attention in , σ represents the softmax operation, BN(·) represents the batch normalization operation, represents element-by-element addition, Represents matrix multiplication; traversal All eigenvalues ​​of Step 11: Calculate the sign language features of each sign language video in the sign language video set V. The sign language video clip features after time channel attention and time channel attention calibration Among them, v i ∈V, ⊙ represents element-by-element multiplication; Step 12: Calculate the coarse-grained features of each sign language video in the sign language video set V Among them, TCP(·) represents temporal covariance pooling, MPN(·) represents matrix power normalization, and FC(·) represents a fully connected layer; Step 13: Calculate the similarity matrix between the fine-grained features of each sign language video in the sign language video set V and the fine-grained features of each text in the text description set T corresponding to the sign language video: Among them, v i ∈V,t j ∈T; Step 14: Calculate the fine-grained similarity matrix from text to sign language video between each sign language video in the sign language video set V and each text in the text description set T corresponding to the sign language video: E t2v (i,j)=E f (i,j)·ColSoftmax(E f (i,j)); Among them, ColSoftmax(·) indicates that softmax calculation is performed by column; then the fine-grained similarity score from text to sign language video is expressed as: S t2v (i,j)=Avg(ColSum(E t2v (i,j))); Among them, Avg(·) means finding the average value, and ColSum(·) means summing the columns of the matrix; Step 15: Calculate the fine-grained similarity matrix from sign language video to text between each sign language video in the sign language video set V and each text in the text description set T: E v2t (i,j)=E f (i,j)·RowSoftmax(E f (i,j)); Then the fine-grained similarity score from sign language video to text is expressed as: S v2t (i,j)=Avg(RowSum(E v2t (i,j))); Among them, RowSum(·) means the sum of the rows of the matrix; Step 16: Calculate the coarse-grained similarity score between each sign language video in the sign language video set V and each text in the text description set T: Among them, || || represents the modulo operation, Indicates sign language video v i ∈V, a coarse-grained video feature. Represents text t j ∈T coarse-grained sentence features; Step 17: Calculate the coarse-grained similarity score between each sign language video in the sign language video set V and each text in the text description set T: Step 18: Calculate the InfoNCE loss function for fine-grained alignment of text and sign language video Where τ is the learnable temperature coefficient and B represents the number of video-text pairs in the same batch; Step 19: Calculate the InfoNCE loss function for the coarse-grained alignment of text and sign language video. Step 20: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each sign language video in the sign language video set V: Among them, v i ∈V, Attn(Q,K,V) is the dot product attention, μ and σ represent The mean vector and variance vector of the feature distribution, F(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution: p(z|v)~N(μ,diag(σ 2 )): Step 21: Use the shared attention residual Gaussian distribution network to obtain the mean and variance of the fine-grained feature distribution of each text in the text description set T: Among them, t j ∈T, Δμ and Δσ represent The feature distribution relative to The relative position and relative scale of feature distribution, G(·) represents the linear network layer; then Parameterized as a multivariate Gaussian distribution: p(z|v,t)~N(μ+Δμ,diag(σ 2 ·Board 2 )); Step 22: Calculate the KL divergence loss based on the same noise sampled from standard Gaussian noise and reparameterization, L KL =KL(p(z|v,t)||p(z|v)); Step 23: Calculate the overall loss function. L=L f +αL c +βL s +λL KL ; Among them, α is the ratio of coarse-grained alignment loss, β is the ratio of video similarity loss, and λ is the ratio of Gaussian distribution alignment loss; Step 24: Repeat steps 4 to 23. When L is less than the specified minimum loss value, the calculation ends. Step 25: For the query text, select the sign language video with the highest fine-grained similarity score from the text to the sign language video as the retrieval result.

Images