A continuous sign language recognition method based on multi-granularity cross-modal contrastive learning

Abstract

A continuous sign language recognition method based on multi-granularity cross-modal contrastive learning first fully exploits the implicit semantic mapping relationship between sign language videos and corresponding sign language texts. It then utilizes a multi-granularity cross-modal contrast mechanism to mitigate the potential impact of redundant frames and semantically irrelevant identifiers on cross-modal alignment. Through coarse-grained video-sentence contrast and fine-grained frame-word contrast strategies, it effectively aligns the features of visual and textual modalities from both a global and local perspective. Cross-granularity video-word contrast and cross-granularity frame-sentence contrast strategies mitigate the semantic interference caused by redundant frames and semantically irrelevant identifiers. Finally, the method utilizes the visual embeddings obtained in the visual encoder and obtains the sign language text sentences through the decoder as the predicted continuous sign language recognition results. This method fully exploits the implicit semantic mapping relationship between sign language videos and corresponding sign language texts, enhancing the sign language model's semantic understanding of visual context and improving the applicability and accuracy of sign language recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of continuous sign language recognition, and in particular to a continuous sign language recognition method based on multi-granularity cross-modal contrastive learning. Background Art

[0002] Sign language is a natural language in a visual-spatial form. People with hearing impairments convey linguistic information through gestures and facial expressions, making it a crucial way for the hearing-impaired to participate in social interaction. Continuous sign language recognition can effectively overcome communication barriers between the hearing-impaired and the rest of society, enhancing social inclusion and interpersonal understanding. Furthermore, continuous sign language recognition can be applied to smart devices, translation software, and other applications, helping the hearing-impaired communicate more naturally with others, thus possessing significant practical significance.

[0003] The shortage of sign language video corpus makes it difficult to build a video feature extractor with good generalization ability based solely on visual information. On the one hand, cross-modal alignment methods indirectly supplement sign language video data by utilizing the additional semantic information provided by sign language annotations, thereby effectively alleviating the problem caused by the shortage of sign language video corpus. On the other hand, visual models can use the semantic information in the text to support the learning of video features, thereby improving the generalization ability of the model. Therefore, more and more researchers have turned their attention to the problem of cross-modal alignment. However, some existing cross-modal continuous sign language recognition methods mainly align the feature representations of visual and text modalities from a video-level or frame-level perspective, but ignore the irrelevant semantic interference caused by redundant frames and identifiers. In some complex sign language video scenes, this redundant irrelevant semantic information will significantly affect the recognition accuracy of sign language and the effect of cross-modal alignment. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, to improve the effect of cross-modal alignment in continuous sign language recognition, and to consider the potential impact of redundant frames and semantically irrelevant identifiers on the final recognition results, the present invention proposes a continuous sign language recognition method based on multi-granularity cross-modal contrastive learning, which utilizes the coarse and fine granularity information of the visual modality and text modality and the multi-granularity cross-modal semantic alignment mechanism to achieve more accurate continuous sign language recognition.

[0005] The technical solution adopted by the present invention to solve the technical problem is:

[0006] A continuous sign language recognition method based on multi-granularity cross-modal contrastive learning first fully exploits the implicit semantic mapping relationship between sign language videos and corresponding sign language texts. Then, a multi-granularity cross-modal contrast mechanism is used to alleviate the potential impact of redundant frames and semantically irrelevant identifiers on cross-modal alignment. Through coarse-grained video-sentence contrast and fine-grained frame-word contrast strategies, the features of visual and text modalities are effectively aligned from both global and local perspectives. The semantic interference caused by redundant frames and semantically irrelevant identifiers is alleviated through cross-granularity video-word contrast and cross-granularity frame-sentence contrast strategies. Finally, the visual embedding obtained from the visual encoder is used to obtain the sign language text sentence through the decoder as the predicted continuous sign language recognition result.

[0007] Furthermore, the method comprises the following steps:

[0008] Step 1: A set of frames of a sign language video is represented as V = {x1, x2…, x n}, where n represents the number of frames in the video, x1 represents the first frame of the video, x2 represents the second frame of the video, and x n represents the nth frame of video, and the text annotation sentence corresponding to the sign language video is denoted as T;

[0009] Step 2: Preprocess the data and perform data enhancement operations such as center cropping, adjusting width and height, etc. on the video frames in the sign language video data V to obtain the processed video data. At the same time, the sign language text annotation data T is segmented and other operations are performed to obtain the processed text data in Represents the word set of the sign language annotation sentence, m represents the number of words in the sentence, y1 represents the first text label, y i is the i-th text label, y m Represents the mth text label;

[0010] Step 3: Extract using 2D convolutional neural network The initial features of the video frame are then captured using a one-dimensional convolutional neural network and a bidirectional long short-term memory network. The temporal features of adjacent video frames and the dependencies between long-distance frames are used to obtain a fine-grained embedding set of video frames. in is the fine-grained embedding of frame 1, is the fine-grained embedding of the i-th frame, is the fine-grained embedding of the nth frame, and then the pooling operation is used to obtain the coarse-grained embedding representing the overall semantic features of the video

[0011] Step 4: Use the visual decoder to embed fine-grained Convert it into text sequence features, identify the text sequence features, obtain multiple candidate text sequence IDs, and use the Softmax function to select the one with the highest probability as the predicted text ID. Then, use a fully connected layer to map the predicted text ID to the corresponding text t1′ in the vocabulary. Traverse the fine-grained embedding set of video frames and obtain the text corresponding to each fine-grained embedding according to this method, thereby obtaining the predicted sign sentence T predict ={t1′,t′2,…,t′ m};

[0012] Step 5: Calculate the predicted sign language text T predict The loss function between the real sign language text T

[0013]

[0014] Among them, L represents the set of all possible text label combination paths π, τ i For the text label y i The probability of appearing in path π, m represents T predict The number of words in

[0015] Step 6: Use CLIP text encoder to extract text data Extract fine-grained embeddings representing the features of each sign language word Then, the multi-layer self-attention mechanism of the Transformer model is used to capture the relationship between different words and their contextual information, and the pooling operation is used to obtain a coarse-grained embedding that represents the overall semantic features of the entire sentence.

[0016] Step 7: Calculate the coarse-grained comparison similarity score between video and text

[0017]

[0018] Step 8: Calculate cross-granularity frame and sentence comparison similarity scores

[0019]

[0020] Step 9: Calculate cross-granularity video and word comparison similarity scores

[0021]

[0022] Step 10: Calculate the fine-grained comparative similarity score between video and text

[0023]

[0024] in

[0025]

[0026] Step 11: Calculate the InfoNCE loss function for the similarity score between the sign language video and the corresponding text annotation

[0027] L sim =InfoNCE(Softmax(Sim V-S ,Sim F-S ,Sim V-W ,Sim F-W )

[0028] *{Sim V-S ,Sim F-S ,Sim V-W ,Sim F-W})

[0029] Step 12: Calculate the total loss function L all =L ctc +λ*L sim , where λ is used to balance L ctc and L sim Hyperparameters of

[0030] Step 13: Calculate word error rate

[0031]

[0032] Among them, #ins represents the number of words that appear in the predicted text sign language but do not exist in the real sign language text, #del represents the number of words that appear in the real sign language text but are not recognized in the predicted text sign language, #sub represents the number of words that appear in the real sign language text but are incorrectly recognized as other words in the predicted text sign language, and #reference represents the total number of words in the real sign language text.

[0033] Step 14: Repeat steps 3 to 13. all When it drops to the specified threshold, the calculation ends and the final predicted sign language text T predict As the result of sign language recognition.

[0034] The technical concept of the present invention is as follows: In the continuous sign language recognition method, the implicit semantic mapping relationship between the sign language video and the corresponding sign language text is fully exploited, and a multi-granularity cross-modal comparison mechanism is used to alleviate the potential impact of redundant frames and semantically irrelevant identifiers on cross-modal alignment, thereby improving the accuracy of sign language recognition.

[0035] The beneficial effects of the present invention are: it can alleviate the potential impact of redundant frames and semantically irrelevant words in the cross-modal alignment process, enhance the model's semantic understanding of visual context, and improve the applicability and accuracy of sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the flow of continuous sign language recognition method based on multi-granularity cross-modal contrastive learning. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to the accompanying drawings.

[0038] Reference Figure 1 A continuous sign language recognition method based on multi-granularity cross-modal contrastive learning is proposed. This method fully exploits the implicit semantic mapping relationship between sign language videos and corresponding sign language texts, enhances the sign language model's semantic understanding of visual context, and improves the applicability and accuracy of sign language recognition. The method includes the following steps:

[0039] Step 1: A set of frames of a sign language video is represented as V = {x1, x2…, x n}, where n represents the number of frames in the video, x1 represents the first frame of the video, x2 represents the second frame of the video, and x n represents the nth frame of video, and the text annotation sentence corresponding to the sign language video is denoted as T;

[0040] Step 2: Preprocess the data and perform data enhancement operations such as center cropping, adjusting width and height, etc. on the video frames in the sign language video data V to obtain the processed video data. At the same time, the sign language text annotation data T is segmented and other operations are performed to obtain the processed text data in Represents the word set of the sign language annotation sentence, m represents the number of words in the sentence, y1 represents the first text label, y i is the i-th text label, y m Represents the mth text label;

[0041] Step 3: Extract using 2D convolutional neural network The initial features of the video frame are then captured using a one-dimensional convolutional neural network and a bidirectional long short-term memory network. The temporal features of adjacent video frames and the dependencies between long-distance frames are used to obtain a fine-grained embedding set of video frames. in is the fine-grained embedding of frame 1, is the fine-grained embedding of the i-th frame, is the fine-grained embedding of the nth frame, and then the pooling operation is used to obtain the coarse-grained embedding representing the overall semantic features of the video

[0042] Step 4: Use the visual decoder to embed fine-grained Convert it into text sequence features, identify the text sequence features, obtain multiple candidate text sequence IDs, and use the Softmax function to select the one with the highest probability as the predicted text ID. Then, use a fully connected layer to map the predicted text ID to the corresponding text t1′ in the vocabulary. Traverse the fine-grained embedding set of video frames and obtain the text corresponding to each fine-grained embedding according to this method, thereby obtaining the predicted sign sentence T predict ={t1′,t′2,…,t′ m};

[0043] Step 5: Calculate the predicted sign language text T predict The loss function between the real sign language text T

[0044]

[0045] Among them, L represents the set of all possible text label combination paths π, τ i For the text label y i The probability of appearing in path π, m represents T predict The number of words in

[0046] Step 6: Use CLIP text encoder to extract text data Extract fine-grained embeddings representing the features of each sign language word Then, the multi-layer self-attention mechanism of the Transformer model is used to capture the relationship between different words and their contextual information, and the pooling operation is used to obtain a coarse-grained embedding that represents the overall semantic features of the entire sentence.

[0047] Step 7: Calculate the coarse-grained comparison similarity score between video and text

[0048]

[0049] Step 8: Calculate cross-granularity frame and sentence comparison similarity scores

[0050]

[0051] Where n is the set The number of frames is the same as the number of frames in the video in step 1;

[0052] Step 9: Calculate cross-granularity video and word comparison similarity scores

[0053]

[0054] Where m is a set The number of words in the sentence is the same as the number of words in step 2;

[0055] Step 10: Calculate the fine-grained comparative similarity score between video and text

[0056]

[0057] in

[0058]

[0059] Step 11: Calculate the InfoNCE loss function for the similarity score between the sign language video and the corresponding text annotation

[0060] L sim =InfoNCE(Softmax(Sim V-S ,Sim F-S ,Sim V-W ,Sim F-W )

[0061] *{Sim V-S ,Sim F-S ,Sim V-W ,Sim F-W})

[0062] Step 12: Calculate the total loss function L all =L ctc +λ*L sim , where λ is used to balance L ctc and L sim Hyperparameters of

[0063] Step 13: Calculate word error rate

[0064]

[0065] Among them, #ins represents the number of words that appear in the predicted text sign language but do not exist in the real sign language text, #del represents the number of words that appear in the real sign language text but are not recognized in the predicted text sign language, #sub represents the number of words that appear in the real sign language text but are incorrectly recognized as other words in the predicted text sign language, and #reference represents the total number of words in the real sign language text;

[0066] Step 14: Repeat steps 3 to 13. all When it drops to the specified threshold, the calculation ends and the final predicted sign language text T predict As the result of sign language recognition.

[0067] In this embodiment, for the continuous sign language recognition method, given a set of frames of a sign language video, the corresponding real sign language text label is "<start identifier>Cannot drink while driving<end identifier>", where the identifier does not have semantic information. Through preprocessing, the video frame is subjected to data enhancement operations such as central character amplification and noise addition, and the sign language label is segmented to obtain a word set = {drive, cannot, drink}. Then, the visual and text encoders are used to obtain coarse and fine granularity embeddings of the sign language video and text respectively. The similarity scores between the two are calculated through a multi-granularity cross-modal comparison mechanism, and the multi-granularity comparison loss L is obtained using the InfoNCE function. sim , in order to mine the implicit semantic mapping relationship between the sign language video and the corresponding sign language text, the visual embedding enters the decoder to obtain the predicted sign language text sentence, and then calculates it with the real sign language text label to obtain the loss L ctc , the parameters of the visual encoder and decoder are reversely updated through the loss, so that after the visual embedding from the encoder passes through the decoder, the video frame can be correctly identified as "Don't drink and drive".

[0068] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by persons of ordinary skill in the art based on the invention.

Claims

1. A continuous sign language recognition method based on multi-granularity cross-modal contrastive learning, characterized by: First, the implicit semantic mapping relationship between sign language videos and corresponding sign language texts is fully explored. Then, a multi-granular cross-modal comparison mechanism is used to alleviate the potential impact of redundant frames and semantically irrelevant identifiers on cross-modal alignment. Through coarse-grained video-sentence comparison and fine-grained frame-word comparison strategies, the features of visual and text modalities are effectively aligned from both global and local perspectives. The semantic interference caused by redundant frames and semantically irrelevant identifiers is alleviated through cross-granular video-word comparison and cross-granular frame-sentence comparison strategies. Finally, the visual embedding obtained in the visual encoder is used to obtain the sign language text sentence through the decoder as the predicted continuous sign language recognition result. The method comprises the following steps: Step 1: A frame set of a sign language video is represented as ,in Represents the number of frames in the video, Indicates the first frame of video, Indicates the second frame of video, Indicates the Frame video, the text annotation sentence corresponding to the sign language video is expressed as ; Step 2: Preprocess the data and process the sign language video data Perform data enhancement operations such as center cropping and adjusting width and height on the video frames to obtain processed video data , while annotating the sign language text data Perform operations such as word segmentation to obtain processed text data ,in , represents the word set of the sign language annotation sentence, Represents the number of words in a sentence, Indicates the first text label, For the text labels, Indicates the text labels; Step 3: Extract using 2D convolutional neural network The initial features of the video frame are then captured using a one-dimensional convolutional neural network and a bidirectional long short-term memory network. The temporal features of adjacent video frames and the dependencies between long-distance frames are used to obtain a fine-grained embedding set of video frames. ,in is the fine-grained embedding of frame 1, For the Fine-grained embedding of frames, For the The fine-grained embedding of the frame is then used to obtain the coarse-grained embedding that represents the overall semantic features of the video. ; Step 4: Use the visual decoder to embed fine-grained Convert it into text sequence features, identify the text sequence features, obtain multiple candidate text sequence IDs, and use the Softmax function to select the one with the highest probability as the predicted text ID. Then map the predicted text ID to the corresponding text in the vocabulary through a fully connected layer. ; Traverse the fine-grained embedding set of the video frame, and obtain the text corresponding to each fine-grained embedding according to this method, so as to obtain the predicted sign sentence ; Step 5: Calculate the predicted sign language text With real sign language text The loss function between ； in, Represents all possible text label combination paths A collection of For text labels Appears on the path The probability inside, represent The number of words in Step 6: Use CLIP text encoder to extract text data Extract fine-grained embeddings representing the features of each sign language word Then, the multi-layer self-attention mechanism of the Transformer model is used to capture the relationship between different words and their contextual information, and the pooling operation is used to obtain a coarse-grained embedding that represents the overall semantic features of the entire sentence. ; Step 7: Calculate the coarse-grained comparison similarity score between video and text ； Step 8: Calculate cross-granularity frame and sentence comparison similarity scores ； Step 9: Calculate cross-granularity video and word comparison similarity scores ； Step 10: Calculate the fine-grained comparative similarity score between video and text ； in ； ； Step 11: Calculate the InfoNCE loss function for the similarity score between the sign language video and the corresponding text annotation ； Step 12: Calculate the total loss function ,in It is used for balance and Hyperparameters of Step 13: Calculate word error rate ； in, #ins Indicates the number of words that appear in the predicted text sign language but do not exist in the real sign language text, # del Indicates the number of words that appear in the real sign language text but are not recognized in the predicted text sign language. #sub Indicates the number of words that appear in the real sign language text but are incorrectly recognized as other words in the predicted text sign language. #reference represents the total number of words in the real sign language text; Step 14: Repeat steps 3 to 13. When it drops to the specified threshold, the calculation ends and the final predicted sign language text is As the result of sign language recognition.

Images