Continuous sign language recognition method based on multi-granularity cross-modal comparative learning

Abstract

The invention discloses a continuous sign language recognition method based on multi-granularity cross-modal comparative learning. The method comprises the following steps: firstly, fully tapping a hidden semantic mapping relationship between a hidden sign language video and a corresponding sign language text; then, a multi-granularity cross-modal comparison mechanism is used for relieving potential influences of redundant frames and semantic irrelevant identification words on cross-modal alignment, and through a coarse-granularity video-sentence comparison strategy and a fine-granularity frame-word comparison strategy, visual and text modal features are effectively aligned from global and local angles; semantic interference caused by redundant frames and semantic irrelevant identification words is relieved through cross-granularity video-word comparison and cross-granularity frame-sentence comparison strategies; and finally, obtaining sign language text sentences through a decoder by utilizing visual embedding obtained in a visual encoder, and taking the sign language text sentences as predicted continuous sign language recognition results. The hidden semantic mapping relation between the sign language video and the corresponding sign language text is fully excavated, the semantic understanding of the sign language model on the visual context is enhanced, and the applicability and accuracy of sign language recognition are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of continuous sign language recognition, and in particular to a continuous sign language recognition method based on multi-granularity cross-modal contrastive learning. Background Art

[0002] Sign language is a natural language in the form of visual space. The hearing-impaired convey language information through gestures, facial expressions, etc., which is an important way for the hearing-impaired group to participate in social communication. Continuous sign language recognition can effectively break down the communication barriers between the hearing-impaired group and other groups in society, and enhance social inclusiveness and interpersonal understanding. In addition, continuous sign language recognition can also be applied to smart devices, translation software, etc., to help the hearing-impaired group communicate with others more naturally, which has important practical significance.

[0003] The shortage of sign language video corpus makes it difficult to build a video feature extractor with good generalization ability based solely on visual information. On the one hand, the cross-modal alignment method indirectly supplements the sign language video data by utilizing the additional semantic information provided by sign language annotations, thereby effectively alleviating the problem caused by the shortage of sign language video corpus. On the other hand, the visual model can support the learning of video features with the help of semantic information in the text, thereby improving the generalization ability of the model. Therefore, more and more researchers have turned their attention to the cross-modal alignment problem. However, some existing cross-modal continuous sign language recognition methods mainly align the feature representations of visual and textual modalities from the video level or frame level, but ignore the irrelevant semantic interference caused by redundant frames and identifiers. In some complex sign language video scenes, these redundant irrelevant semantic information will significantly affect the recognition accuracy of sign language and the effect of cross-modal alignment. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, to improve the effect of cross-modal alignment in continuous sign language recognition, and to consider the potential impact of redundant frames and semantically irrelevant identifiers on the final recognition results, the present invention proposes a continuous sign language recognition method based on multi-granularity cross-modal contrastive learning, which utilizes the coarse and fine granularity information of the visual modality and the text modality and the multi-granularity cross-modal semantic alignment mechanism to achieve more accurate continuous sign language recognition.

[0005] The technical solution adopted by the present invention to solve the technical problem is:

[0006] A continuous sign language recognition method based on multi-granularity cross-modal contrastive learning first fully explores the implicit semantic mapping relationship between sign language videos and corresponding sign language texts; then uses a multi-granularity cross-modal contrast mechanism to alleviate the potential impact of redundant frames and semantically irrelevant markers on cross-modal alignment, and effectively aligns the features of visual and textual modalities from a global and local perspective through coarse-grained video-sentence contrast and fine-grained frame-word contrast strategies, and alleviates the semantic interference caused by redundant frames and semantically irrelevant markers through cross-granularity video-word contrast and cross-granularity frame-sentence contrast strategies; finally, the visual embedding obtained in the visual encoder is used to obtain the sign language text sentence through the decoder as the predicted continuous sign language recognition result.

[0007] Further, the method comprises the following steps:

[0008] Step 1: A frame set of a sign language video is represented as V = {x1, x2…, x n}, where n represents the number of frames in the video, x1 represents the first frame of the video, x2 represents the second frame of the video, and x n represents the nth frame of video, and the text annotation sentence corresponding to the sign language video is denoted as T;

[0009] Step 2: Perform data preprocessing, perform data enhancement operations such as center cropping, adjusting width and height on the video frames in the sign language video data V, and obtain processed video data At the same time, the sign language text annotation data T is segmented and other operations are performed to obtain the processed text data in represents the word set of the sign language annotation sentence, m represents the number of words in the sentence, y1 represents the first text label, y i is the i-th text label, y m Represents the mth text label;

[0010] Step 3: Extract using 2D convolutional neural network The initial features of the video frame are then captured using a one-dimensional convolutional neural network and a bidirectional long short-term memory network. The temporal features of adjacent video frames and the dependencies between long-distance frames are used to obtain a fine-grained embedding set of video frames. in is the fine-grained embedding of frame 1, is the fine-grained embedding of the i-th frame, is the fine-grained embedding of the nth frame, and then the pooling operation is used to obtain the coarse-grained embedding representing the overall semantic features of the video

[0011] Step 4: Use the visual decoder to embed fine-grained Convert it into text sequence features, identify the text sequence features, obtain multiple candidate text sequence IDs, and use the Softmax function to select the one with the highest probability as the predicted text ID. Map the predicted text ID to the corresponding text t1′ in the vocabulary through a fully connected layer; traverse the fine-grained embedding set of video frames, and obtain the text corresponding to each fine-grained embedding according to this method, so as to obtain the predicted sign sentence T predict ={t1′,t′2,…,t′ m};

[0012] Step 5: Calculate the predicted sign language text T predict The loss function between the real sign language text T

[0013]

[0014] Among them, L represents the set of all possible text label combination paths π, τ i For the text label y i The probability of appearing in path π, m represents T predict The number of words in

[0015] Step 6: Use CLIP text encoder to extract text data Extract fine-grained embeddings representing the features of each sign language word Then, the multi-layer self-attention mechanism of the Transformer model is used to capture the relationship between different words and their contextual information, and the pooling operation is used to obtain a coarse-grained embedding that represents the overall semantic features of the entire sentence.

[0016] Step 7: Calculate the coarse-grained comparison similarity score between video and text

[0017]

[0018] Step 8: Calculate cross-granularity frame and sentence comparison similarity scores

[0019]

[0020] Step 9: Calculate cross-granularity video and word comparison similarity scores

[0021]

[0022] Step 10: Calculate the fine-grained comparative similarity scores of video and text

[0023]

[0024] in

[0025]

[0026] Step 11: Calculate the InfoNCE loss function of the similarity score between the sign language video and the corresponding text annotation

[0027] L sim =InfoNCE(Softmax(Sim V-S ,Sim F-S ,Sim V-W ,Sim F-W )

[0028] *{Sim V-S ,Sim F-S ,Sim V-W ,Sim F-W})

[0029] Step 12: Calculate the total loss function L all =L ctc +λ*L sim , where λ is used to balance L ctc and L sim Hyperparameters of

[0030] Step 13: Calculate word error rate

[0031]

[0032] Among them, #ins represents the number of words that appear in the predicted text sign language but do not exist in the real sign language text, #del represents the number of words that appear in the real sign language text but are not recognized in the predicted text sign language, #sub represents the number of words that appear in the real sign language text but are mistakenly recognized as other words in the predicted text sign language, and #reference represents the total number of words in the real sign language text.

[0033] Step 14: Repeat steps 3 to 13. all When it drops to the specified threshold, the calculation ends and the final predicted sign language text T predict As the result of sign language recognition.

[0034] The technical concept of the present invention is as follows: In the continuous sign language recognition method, the implicit semantic mapping relationship between the sign language video and the corresponding sign language text is fully explored, and a multi-granularity cross-modal comparison mechanism is used to alleviate the potential impact of redundant frames and semantically irrelevant identifiers on cross-modal alignment, thereby improving the accuracy of sign language recognition.

[0035] The beneficial effects of the present invention are: being able to alleviate the potential impact of redundant frames and semantically irrelevant words in the cross-modal alignment process, enhancing the model's semantic understanding of visual context, and improving the applicability and accuracy of sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the flow of the continuous sign language recognition method based on multi-granularity cross-modal contrastive learning. DETAILED DESCRIPTION

[0037] The present invention is further described below in conjunction with the accompanying drawings.

[0038] Reference Figure 1 A continuous sign language recognition method based on multi-granularity cross-modal contrastive learning is proposed. The method fully exploits the implicit semantic mapping relationship between sign language videos and corresponding sign language texts, enhances the semantic understanding of visual context by the sign language model, and improves the applicability and accuracy of sign language recognition. The method comprises the following steps:

[0039] Step 1: A frame set of a sign language video is represented as V = {x1, x2…, x n}, where n represents the number of frames in the video, x1 represents the first frame of the video, x2 represents the second frame of the video, and x n represents the nth frame of video, and the text annotation sentence corresponding to the sign language video is denoted as T;

[0040] Step 2: Perform data preprocessing, perform data enhancement operations such as center cropping, adjusting width and height on the video frames in the sign language video data V, and obtain processed video data At the same time, the sign language text annotation data T is segmented and other operations are performed to obtain the processed text data in represents the word set of the sign language annotation sentence, m represents the number of words in the sentence, y1 represents the first text label, y i is the i-th text label, y m Represents the mth text label;

[0041] Step 3: Extract using 2D convolutional neural network The initial features of the video frame are then captured using a one-dimensional convolutional neural network and a bidirectional long short-term memory network. The temporal features of adjacent video frames and the dependencies between long-distance frames are used to obtain a fine-grained embedding set of video frames. in is the fine-grained embedding of frame 1, is the fine-grained embedding of the i-th frame, is the fine-grained embedding of the nth frame, and then the pooling operation is used to obtain the coarse-grained embedding representing the overall semantic features of the video

[0042] Step 4: Use the visual decoder to embed fine-grained Convert it into text sequence features, identify the text sequence features, obtain multiple candidate text sequence IDs, and use the Softmax function to select the one with the highest probability as the predicted text ID. Map the predicted text ID to the corresponding text t1′ in the vocabulary through a fully connected layer; traverse the fine-grained embedding set of video frames, and obtain the text corresponding to each fine-grained embedding according to this method, so as to obtain the predicted sign sentence T predict ={t1′,t′2,…,t′ m};

[0043] Step 5: Calculate the predicted sign language text T predict The loss function between the real sign language text T

[0044]

[0045] Among them, L represents the set of all possible text label combination paths π, τ i For the text label y i The probability of appearing in path π, m represents T predict The number of words in

[0046] Step 6: Use CLIP text encoder to extract text data Extract fine-grained embeddings representing the features of each sign language word Then, the multi-layer self-attention mechanism of the Transformer model is used to capture the relationship between different words and their contextual information, and the pooling operation is used to obtain a coarse-grained embedding that represents the overall semantic features of the entire sentence.

[0047] Step 7: Calculate the coarse-grained comparison similarity score between video and text

[0048]

[0049] Step 8: Calculate cross-granularity frame and sentence comparison similarity scores

[0050]

[0051] Where n is the set The number of frames is the same as the number of frames in the video in step 1;

[0052] Step 9: Calculate cross-granularity video and word comparison similarity scores

[0053]

[0054] Where m is a set The number of words in the sentence is the same as the number of words in step 2;

[0055] Step 10: Calculate the fine-grained comparative similarity scores of video and text

[0056]

[0057] in

[0058]

[0059] Step 11: Calculate the InfoNCE loss function of the similarity score between the sign language video and the corresponding text annotation

[0060] L sim =InfoNCE(Softmax(Sim V-S ,Sim F-S ,Sim V-W ,Sim F-W )

[0061] *{Sim V-S ,Sim F-S ,Sim V-W ,Sim F-W})

[0062] Step 12: Calculate the total loss function L all =L ctc +λ*L sim , where λ is used to balance L ctc and L sim Hyperparameters of

[0063] Step 13: Calculate word error rate

[0064]

[0065] Among them, #ins represents the number of words that appear in the predicted text sign language but do not exist in the real sign language text, #del represents the number of words that appear in the real sign language text but are not recognized in the predicted text sign language, #sub represents the number of words that appear in the real sign language text but are mistakenly recognized as other words in the predicted text sign language, and #reference represents the total number of words in the real sign language text;

[0066] Step 14: Repeat steps 3 to 13. all When it drops to the specified threshold, the calculation ends and the final predicted sign language text T predict As the result of sign language recognition.

[0067] In this embodiment, for the continuous sign language recognition method, a frame set of a sign language video is given, and its corresponding real sign language text label is "<start identifier>Don't drink while driving<end identifier>", where the identifier does not have semantic information. The video frame is preprocessed to perform data enhancement operations such as central character enlargement and noise addition, and the sign language label is segmented to obtain a word set = {drive, can't, drink}. Then, the coarse and fine granularity embedding of the sign language video and text is obtained respectively by using the visual and text encoders, and the similarity scores between the two are calculated by the multi-granularity cross-modal comparison mechanism, and the multi-granularity comparison loss L is obtained by using the InfoNCE function. sim , in order to mine the implicit semantic mapping relationship between the sign language video and the corresponding sign language text, the visual embedding enters the decoder to obtain the predicted sign language text sentence, and then calculates it with the real sign language text label to obtain the loss L ctc , the loss is used to reversely update the parameters of the visual encoder and decoder, so that after the visual embedding from the encoder passes through the decoder, the video frame can be correctly identified as "Don't drink and drive".

[0068] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A continuous sign language recognition method based on multi-granularity cross-modal contrastive learning, characterized in that: First, the implicit semantic mapping relationship between sign language videos and the corresponding sign language texts is fully explored; then, a multi-granularity cross-modal comparison mechanism is used to alleviate the potential impact of redundant frames and semantically irrelevant markers on cross-modal alignment. Through coarse-grained video-sentence comparison and fine-grained frame-word comparison strategies, the features of visual and textual modalities are effectively aligned from global and local perspectives. Cross-granularity video-word comparison and cross-granularity frame-sentence comparison strategies are used to alleviate the semantic interference caused by redundant frames and semantically irrelevant markers; finally, the visual embedding obtained in the visual encoder is used to obtain the sign language text sentence through the decoder as the predicted continuous sign language recognition result.

2. The method for continuous sign language recognition based on multi-granularity cross-modal contrastive learning as claimed in claim 1, characterized in that: The method comprises the following steps: Step 1: A frame set of a sign language video is represented as V = {x1, x2…, x n }, where n represents the number of frames in the video, x1 represents the first frame of the video, x2 represents the second frame of the video, and x n represents the nth frame of video, and the text annotation sentence corresponding to the sign language video is denoted as T; Step 2: Perform data preprocessing, perform data enhancement operations such as center cropping, adjusting width and height on the video frames in the sign language video data V, and obtain processed video data At the same time, the sign language text annotation data T is segmented and other operations are performed to obtain the processed text data in represents the word set of the sign language annotation sentence, m represents the number of words in the sentence, y1 represents the first text label, y i is the i-th text label, y m Represents the mth text label; Step 3: Extract using 2D convolutional neural network The initial features of the video frame are then captured using a one-dimensional convolutional neural network and a bidirectional long short-term memory network. The temporal features of adjacent video frames and the dependencies between long-distance frames are used to obtain a fine-grained embedding set of video frames. in is the fine-grained embedding of frame 1, is the fine-grained embedding of the i-th frame, is the fine-grained embedding of the nth frame, and then the pooling operation is used to obtain the coarse-grained embedding representing the overall semantic features of the video Step 4: Use the visual decoder to embed fine-grained Convert it into text sequence features, identify the text sequence features, obtain multiple candidate text sequence IDs, and use the Softmax function to select the one with the highest probability as the predicted text ID. Map the predicted text ID to the corresponding text t′1 in the vocabulary through a fully connected layer; traverse the fine-grained embedding set of video frames, and obtain the text corresponding to each fine-grained embedding according to this method, so as to obtain the predicted sign sentence T predict ={t′1,t′2,…,t′ m }; Step 5: Calculate the predicted sign language text T predict The loss function between the real sign language text T Among them, L represents the set of all possible text label combination paths π, τ i For the text label y i The probability of appearing in path π, m represents T predict The number of words in Step 6: Use CLIP text encoder to extract text data Extract fine-grained embeddings representing the features of each sign language word Then, the multi-layer self-attention mechanism of the Transformer model is used to capture the relationship between different words and their contextual information, and the pooling operation is used to obtain a coarse-grained embedding that represents the overall semantic features of the entire sentence. Step 7: Calculate the coarse-grained comparison similarity score between video and text Step 8: Calculate cross-granularity frame and sentence comparison similarity scores Step 9: Calculate cross-granularity video and word comparison similarity scores Step 10: Calculate the fine-grained comparative similarity scores of video and text in Step 11: Calculate the InfoNCE loss function of the similarity score between the sign language video and the corresponding text annotation L sim =InfoNCE(Softmax(Yes V-S ,Yes F-S ,Yes V-W ,Yes F-W ) *{Yes V-S ,Yes F-S ,Yes V-W ,Yes F-W }) Step 12: Calculate the total loss function L all =L ctc +λ*L sim , where λ is used to balance L ctc and L sim Hyperparameters of Step 13: Calculate word error rate Among them, #ins represents the number of words that appear in the predicted text sign language but do not exist in the real sign language text, #del represents the number of words that appear in the real sign language text but are not recognized in the predicted text sign language, #sub represents the number of words that appear in the real sign language text but are mistakenly recognized as other words in the predicted text sign language, and #reference represents the total number of words in the real sign language text. Step 14: Repeat steps 3 to 13. all When it drops to the specified threshold, the calculation ends and the final predicted sign language text T predict As the result of sign language recognition.

Images