Multi-modal continuous sign language recognition method based on comparative learning and related device

Abstract

The invention discloses a multi-modal continuous sign language recognition method and related device based on comparative learning, and the method comprises the steps: inputting multi-modal data into a modal feature enhancement module in a sign language recognition model for feature extraction and enhancement processing, and obtaining enhanced feature data; inputting the enhanced feature data into a comparative learning module in the sign language recognition model for comparative learning processing to obtain a comparative learning result; and inputting the comparison learning result into a feature fusion module in the sign language recognition model, dynamically allocating weights for different features based on a weight generator in the feature fusion module, performing fusion processing, and outputting a sign language recognition prediction result. In the embodiment of the invention, high-level abstract features and low-level time sequence semantics can be extracted from the sign language video, and more comprehensive multi-modal fusion features are formed through the comparative learning and dynamic weight fusion module, so that the accuracy of continuous sign language recognition is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multimodal continuous sign language recognition method and related apparatus based on contrastive learning. Background Technology

[0002] Continuous Sign Language Recognition (CSLR) is a task that uses computer technology to automatically translate sign language videos into text or speech. Biologically speaking, the brain's neural network processing for sign language comprehension begins with the decoding of visual gesture signals in the occipital and posterior temporal lobes. After a series of processing steps in the posterior temporal and parietal regions, lexical semantic representations are activated, and finally, the anterior temporal lobe network, which shares the same network as spoken language, completes the semantic-syntactic combination. This cognitive process can utilize existing computer knowledge (semantic) extraction techniques to assist in perceiving visual information and converting it into information that deaf people can understand. This is crucial for bridging the communication gap between deaf people and hearing people.

[0003] To perceive visual sign language information, many methods incorporate textual semantic information to enhance the model's understanding of deep language structures and improve the accuracy of continuous sign language recognition. However, the deep fusion of visual and textual modalities of sign language still faces technical bottlenecks, and the scale of labeled sign language video data is limited, which restricts the generalization ability and semantic reasoning ability of multimodal continuous sign language recognition models. In recent years, although contrastive learning has been widely used in many fields, its application in multimodal continuous sign language recognition is still relatively limited, and it is usually only used as an auxiliary training method and abandoned in the validation and testing stages. This is because CSLR is a very complex task that requires mapping the entire video sequence into a continuous sequence of sign language words. CSLR not only needs to capture detailed static features, such as hand shape and facial expressions, but also needs to focus on dynamic features, such as arm movements and body posture. Meanwhile, due to the scarcity of annotated sign language video data, some methods treat CSLR as a weakly supervised problem, achieving sentence-level annotation of video sequences and their corresponding text, but lacking frame-level alignment supervision. When using cosine similarity for contrastive learning, sentence-level annotation cannot handle sequence alignment issues, and training with the Connectionist Temporal Classification (CTC) Loss function will lead to poor model performance. Therefore, in CSLR, contrastive learning needs to optimize feature representations for the specificities of sign language movements, fully learning the similarities and differences between samples from different modalities, thereby improving the model's accuracy in recognizing complex and continuous sign language movements. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a multimodal continuous sign language recognition method and related device based on contrastive learning, which can extract high-level abstract features and low-level temporal semantics from sign language videos, and form more comprehensive multimodal fusion features through contrastive learning and dynamic weight fusion modules, thereby improving the accuracy of continuous sign language recognition.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a multimodal continuous sign language recognition method based on contrastive learning, the method comprising: Multimodal data is input into the modal feature enhancement module in the sign language recognition model for feature extraction and enhancement processing to obtain enhanced feature data. The sign language recognition model includes a modal feature enhancement module, a contrast learning module, and a feature fusion module. The multimodal data includes text-annotated data and continuous sign language video data. The enhanced feature data is input into the contrastive learning module in the sign language recognition model for contrastive learning processing to obtain the contrastive learning result. The contrastive learning results are input into the feature fusion module of the sign language recognition model. In the feature fusion module, weights are dynamically assigned to different features based on the weight generator and fusion processing is performed to output the sign language recognition prediction result.

[0006] Optionally, the step of inputting multimodal data into the modal feature enhancement module of the sign language recognition model for feature extraction processing to obtain enhanced feature data includes: After the multimodal data is input into the sign language recognition model, the modal feature enhancement module calls the corresponding data encoder to perform feature extraction processing according to the different modal data in the multimodal data, and obtains the first extracted feature data and the second extracted feature data, wherein the first extracted feature data is the feature data corresponding to the text annotation data, and the second extracted feature data is the feature data corresponding to the continuous sign language video data. The first extracted feature data and the second extracted feature data are input into the Adapter unit in the modal feature enhancement module for feature processing to form the first processed feature data and the second processed feature data. The Adapter unit consists of two linear layers and a layer normalization, and a layer normalization and a ReLU activation function are nested between the two linear layers. The first processed feature data and the second processed feature data are input into the dynamic stacked cross-attention unit in the modal feature enhancement module for enhanced processing of feature depth interaction and information fusion to form enhanced feature data.

[0007] Optionally, the step of calling the corresponding data encoder to perform feature extraction processing based on different modal data in the multimodal data in the modal feature enhancement module to obtain first extracted feature data and second extracted feature data includes: In the modal feature enhancement module, the gloss data encoder is called to perform feature extraction processing based on the text annotation data in the multimodal data to obtain the first extracted feature data. The gloss data encoder is a pre-trained multilingual bidirectional encoder used to extract sign language vocabulary feature data. In the modal feature enhancement module, the video data encoder is invoked to perform feature extraction processing based on the continuous sign language video data in the multimodal data to obtain the second extracted feature data. The video data encoder is a pre-trained SlowFastSign network used to extract visual feature data related to sign language.

[0008] Optionally, the step of inputting the first extracted feature data and the second extracted feature data into the Adapter unit of the modal feature enhancement module for feature processing to form first processed feature data and second processed feature data includes: The first extracted feature data is input into the Gloss2Gloss Adapter unit in the modal feature enhancement module for feature processing to form the first processed feature data; The second extracted feature data is input into the Video2Video Adapter unit in the modal feature enhancement module for feature processing to form the second processed feature data.

[0009] Optionally, the step of inputting the first processed feature data and the second processed feature data into the dynamic stacked cross-attention unit in the modal feature enhancement module for feature depth interaction and information fusion enhancement processing to form enhanced feature data includes: After the first processed feature data and the second processed feature data are input into the dynamic stacked cross attention unit in the modal feature enhancement module, the dynamic stacked cross attention unit uses the second processed feature data as the key K and the value V, and the first processed feature data as the query Q. Enhanced feature data is formed by iteratively processing the first processed feature data as query Q and the second processed feature data as key K and value V in the MLP layer and residual connection within the dynamically stacked cross-attention unit.

[0010] Optionally, the step of inputting the enhanced feature data into the contrastive learning module of the sign language recognition model for contrastive learning processing to obtain the contrastive learning result includes: After the enhanced feature data is input into the contrast learning module in the sign language recognition model, the contrast learning module uses the enhanced feature data and the second processed feature data to perform cosine similarity calculation to obtain the cosine similarity calculation result. The cosine similarity calculation result is used for comparison processing within the similarity adapter of the comparison learning module to obtain the comparison learning result.

[0011] Optionally, the step of dynamically assigning weights to different features and performing fusion processing based on a weight generator in the feature fusion module to output sign language recognition prediction results includes: The feature fusion module uses a weight generator to dynamically assign corresponding weight parameters to different features; The feature fusion module uses the weight parameters corresponding to different feature assignments to perform feature fusion processing on the comparative learning results and the second extracted feature data to form fused feature data. The feature fusion module uses the Fusion Adapter FA(·) function to perform feature transformation processing on the fused feature data, forming a sign language recognition prediction result and outputting it.

[0012] In addition, embodiments of the present invention also provide a multimodal continuous sign language recognition device based on contrastive learning, the device comprising: Feature enhancement module: Used to input multimodal data into the modal feature enhancement module in the sign language recognition model for feature extraction and enhancement processing to obtain enhanced feature data. The sign language recognition model includes a modal feature enhancement module, a contrast learning module, and a feature fusion module. The multimodal data includes text-annotated data and continuous sign language video data. Contrast learning module: used to input the enhanced feature data into the contrast learning module in the sign language recognition model for contrast learning processing to obtain contrast learning results; Fusion prediction module: used to input the contrast learning results into the feature fusion module in the sign language recognition model, and dynamically assign weights to different features based on the weight generator and perform fusion processing in the feature fusion module to output the sign language recognition prediction result.

[0013] In addition, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the processor runs a computer program or code stored in the memory to implement the multimodal continuous sign language recognition method as described in any of the above.

[0014] In addition, embodiments of the present invention also provide a computer-readable storage medium for storing a computer program or code, which, when executed by a processor, implements the multimodal continuous sign language recognition method as described above.

[0015] In this embodiment of the invention, high-level abstract features and low-level temporal semantics can be extracted from sign language videos, and more comprehensive multimodal fusion features can be formed through contrastive learning and dynamic weight fusion modules, thereby improving the accuracy of continuous sign language recognition. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the multimodal continuous sign language recognition method based on contrastive learning in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structural composition of the multimodal continuous sign language recognition device based on contrastive learning in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structural composition of the electronic device in an embodiment of the present invention; Figure 4 This is a framework diagram of the sign language recognition model in an embodiment of the present invention; Figure 5 This is a framework diagram of the dynamically stacked cross-attention unit in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the multimodal continuous sign language recognition method based on contrastive learning in an embodiment of the present invention.

[0020] like Figure 1 As shown, a multimodal continuous sign language recognition method based on contrastive learning is described, the method comprising: S101: Input the multimodal data into the modal feature enhancement module in the sign language recognition model for feature extraction and enhancement processing to obtain enhanced feature data. The sign language recognition model includes a modal feature enhancement module, a contrast learning module, and a feature fusion module. The multimodal data includes text annotation data and continuous sign language video data. In a specific implementation of this invention, the step of inputting multimodal data into the modal feature enhancement module of the sign language recognition model for feature extraction processing to obtain enhanced feature data includes: after inputting the multimodal data into the sign language recognition model, the modal feature enhancement module calls the corresponding data encoder to perform feature extraction processing according to different modal data in the multimodal data to obtain first extracted feature data and second extracted feature data, wherein the first extracted feature data is feature data corresponding to text annotation data, and the second extracted feature data is feature data corresponding to continuous sign language video data; the first extracted feature data and the second extracted feature data are input into the Adapter unit of the modal feature enhancement module for feature processing to form first processed feature data and second processed feature data, wherein the Adapter unit consists of two linear layers and a layer normalization, and a layer normalization and a ReLU activation function are nested between the two linear layers; the first processed feature data and the second processed feature data are input into the dynamic stacked cross-attention unit of the modal feature enhancement module for feature depth interaction and information fusion enhancement processing to form enhanced feature data.

[0021] Furthermore, the step of calling the corresponding data encoder to perform feature extraction processing based on different modal data in the multimodal data in the modal feature enhancement module to obtain first extracted feature data and second extracted feature data includes: calling the gloss data encoder to perform feature extraction processing based on the text annotation data in the multimodal data in the modal feature enhancement module to obtain first extracted feature data, wherein the gloss data encoder is a pre-trained multilingual bidirectional encoder used to extract sign language vocabulary feature data; and calling the video data encoder to perform feature extraction processing based on the continuous sign language video data in the multimodal data in the modal feature enhancement module to obtain second extracted feature data, wherein the video data encoder is a pre-trained SlowFastSign network used to extract visual feature data related to sign language.

[0022] Furthermore, the step of inputting the first extracted feature data and the second extracted feature data into the Adapter unit of the modal feature enhancement module for feature processing to form the first processed feature data and the second processed feature data includes: inputting the first extracted feature data into the Gloss2Gloss Adapter unit of the modal feature enhancement module for feature processing to form the first processed feature data; and inputting the second extracted feature data into the Video2Video Adapter unit of the modal feature enhancement module for feature processing to form the second processed feature data.

[0023] Furthermore, the step of inputting the first processed feature data and the second processed feature data into the dynamic stacked cross-attention unit in the modal feature enhancement module for feature depth interaction and information fusion enhancement processing to form enhanced feature data includes: after inputting the first processed feature data and the second processed feature data into the dynamic stacked cross-attention unit in the modal feature enhancement module, the dynamic stacked cross-attention unit uses the second processed feature data as a key K and a value V, and the first processed feature data as a query Q; and iteratively processes the query Q (the first processed feature data) and the key K and value V (the second processed feature data) in the MLP layer and residual connection within the dynamic stacked cross-attention unit to form enhanced feature data.

[0024] Specifically, such as Figure 4 As shown, the framework for multimodal continuous sign language recognition mainly includes Multimodal Feature Enhancement (MFE), Adaptive Contrastive Similarity (ACS), and Dynamic Weight Allocation (DWA).

[0025] Multimodal data is input into the modality feature enhancement module of the sign language recognition model for feature extraction and enhancement. The multimodal data includes text-annotated data (gloss data) and continuous sign language video data (video data). The feature extraction from gloss data and video data differs significantly. Gloss modality primarily focuses on the semantic information of the language, while video modality needs to capture visual information such as hand movements and facial expressions. Given a sign language video V = {v1, v2, ..., v...} with n frames... nGiven a vocabulary G={g1,g2,...,gm} consisting of m-1 glosses and 1 blank, the goal is to predict consecutive handwritten sentences with a glosses in a video using a designed neural network Net(·). Result= {r1,r2,...,r a} The size of m is equal to the number of categories in the sign language dataset gloss plus 1. The addition of 1 is because blank is also included in this collection. ; Gloss Data Encoder: To enhance the semantic richness of text features, a pre-trained multilingual bidirectional encoder (LaBSE, Language-agnostic BERT Sentence Embeddings) is used to extract sign language vocabulary (gloss) features. Specifically, six custom sentence templates are constructed, and each gloss is extracted from the gloss vocabulary and embedded into these six template sentences, generating a set of sentences related to that gloss. These sentences are then input into LaBSE for feature extraction, outputting six feature vectors. These six feature vectors are averaged to obtain a feature vector that accurately reflects the comprehensive semantics of the gloss. These feature representations not only accurately capture the core semantic information of each gloss but also further enrich the semantic connotation of the features through the incorporation of contextual information. The gloss data encoder E... g (·) Extract features for each g in vocabulary G: ; Where F g It is the result of extracting features from the gloss vocabulary through the gloss data encoder. , where 768 is the dimension of the hidden layer.

[0026] Video Data Encoder: A pre-trained SlowFastSign network is used as the video feature encoder to extract visual features related to sign language. Specifically, a single sign language video is input, processed through SlowFastSign's bidirectional feature fusion module and path feature enhancement module, to obtain the visual feature output. It's important to note that only the feature enhancement part of the path feature enhancement module is used, while its classifier layer is removed, to ensure that the video encoder's output can be directly used for subsequent feature fusion and contrastive learning tasks. This design fully utilizes the SlowFastSign architecture to capture the spatiotemporal information of the video, providing high-quality visual feature representations for continuous sign language recognition tasks. Given a video encoder... : ; in It is the result of extracting features from sign language videos through a video encoder. , where 1024 represents the dimension of the hidden layer.

[0027] The modal feature enhancement module contains Adapter units, which are used to improve the diversity of feature samples. Each Adapter unit consists of two linear layers and a layer normalization function (LayerNorm), with a layer normalization function and a ReLU activation function nested between the two linear layers. The Adapter units used all have the same architecture and are additionally named Gloss2Gloss (G2G) Adapter, Video2Video Adapter, Similarity Adapter, and FusionAdapter. Given features... Where D represents the hidden layer dimension, through linear layers Layer normalization function ReLU activation function and residuals Where 1024 is the dimension of the hidden layer: ; To effectively integrate features from both text and video modalities, a Modality Feature Enhancement (MFE) module is proposed. This module extracts gloss and video features separately using two dedicated encoders and ensures the consistency and complementarity of the two modal features through an adaptive adjustment mechanism. In addition, a Dynamic Stacked Cross-Attention (DSCA) module is designed to further enhance the semantic representation of the gloss features by using the cross-attention mechanism as the core.

[0028] In the modal feature enhancement process, the Gloss2Gloss (G2G) Adapter and Video2Video (V2V) Adapter are introduced to align gloss features with video features. This adaptive module adjusts the feature scale to ensure that gloss and video features maintain dimensionality consistency, thus providing more consistent input for subsequent feature fusion. Furthermore, the adaptive module fine-tunes the outputs of the gloss encoder and video encoder, further optimizing feature representation and improving the model's ability to integrate features from different modalities. The gloss features output by the gloss encoder... The calibrated gloss features are obtained after gloss2gloss Adapter G2G(·). Meanwhile, the video features output by the video encoder The calibrated video features are obtained after Video2VideoAdapter V2V(·). : ; ; like Figure 5 The diagram shows the framework of the Dynamic Stacked Cross-Attention (DSCA) unit. DSCA is proposed to achieve deep interaction and information fusion between gloss features and video features, thereby enriching the gloss features. This unit uses video features F... video As keys and values, gloss features As a query, the input is passed to the cross-attention mechanism, then through the MLP layer and residual connections. The residual output serves as the query, while video features remain as keys and values. Further semantic information is extracted through cross-attention, MLP, and residual connections. After N iterations, the iterative output is used by an α generator to generate flexible weight values. Finally, the original input gloss features and the iterative output are effectively fused, allowing the gloss features to extract rich contextual information from the video features and enhancing their semantic representation. The MLP layer consists of two linear layers with a QuickGELU activation function embedded between them. The iteration number M in the dynamically stacked cross-attention is set to 2. Given the features output by Gloss2GlossAdapter... Features of Video2VideoAdapter output Enhanced feature data can be obtained through dynamic stacking of cross attention. : ; ; ; ; ; in (·) represents the layer normalization function. (·) denotes the normalized exponential function. (·) indicates projection mapping. (·) is one of the activation functions. (·) is the weight generator.

[0029] S102: Input the enhanced feature data into the contrast learning module in the sign language recognition model for contrast learning processing to obtain the contrast learning result; In a specific implementation of this invention, the step of inputting the enhanced feature data into the contrast learning module of the sign language recognition model for contrast learning processing to obtain a contrast learning result includes: after inputting the enhanced feature data into the contrast learning module of the sign language recognition model, the contrast learning module uses the enhanced feature data and the second processed feature data to perform cosine similarity calculation processing to obtain a cosine similarity calculation result; and then, within the similarity adapter of the contrast learning module, the cosine similarity calculation result is used for comparison processing to obtain a contrast learning result.

[0030] Specifically, through modal feature enhancement, not only were semantically rich gloss features obtained (significantly enhanced by video features), but also accurate video features were extracted. Subsequently, the features of these two modalities were normalized, and their cosine similarity was calculated. This process allows us to determine which gloss in the gloss vocabulary is closest to the feature in each temporal dimension of the video, thus providing accurate semantic mapping for continuous sign language recognition. During experiments, it was found that relying solely on cosine similarity as the output resulted in very poor model training results. Therefore, a similarity adapter module was introduced. This module, structurally similar to other adaptive modules and with a simple architecture, significantly improves model performance, allowing gloss features and video features to play a better role at the similarity level. Given video features… and semantically rich gloss features ,calculate and Cosine similarity between And through the Similarity Adapter module Ideal comparison results can be obtained. : ; ; in This represents the magnitude of the eigenvector.

[0031] S103: Input the contrast learning result into the feature fusion module of the sign language recognition model, and dynamically assign weights to different features based on the weight generator and perform fusion processing in the feature fusion module to output the sign language recognition prediction result.

[0032] In a specific implementation of this invention, the step of dynamically assigning weights to different features and performing fusion processing based on a weight generator in the feature fusion module to output sign language recognition prediction results includes: the feature fusion module dynamically assigning corresponding weight parameters to different features using a weight generator; the feature fusion module using the corresponding weight parameters assigned to different features to perform feature fusion processing with the contrast learning results and the second extracted feature data to form fused feature data; and the feature fusion module using the Fusion Adapter FA(·) function to perform feature transformation processing on the fused feature data to form sign language recognition prediction results and output them.

[0033] Specifically, the Dynamic Weight Allocation (DWA) module performs temporal residual fusion on the spatial embeddings of multimodal sign language to facilitate the network's extraction of higher-order abstract features while avoiding the vanishing gradient problem. The core function of DWA is to dynamically assign weights to different features through a weight generator, thereby optimizing the feature fusion process and ultimately outputting classification results. The weight generator can dynamically adjust the relative importance between different features, thus promoting effective feature fusion. The weight generator consists of two linear layers, nested with a LayerNorm and a ReLU activation function. After processing by these layers, the output is further processed by a sigmoid activation function to generate flexible weight values. These weights can be dynamically adjusted according to the characteristics of the input features, ensuring the efficiency and adaptability of the feature fusion process.

[0034] Specifically, one feature is the feature from the output of the video encoder. These features are extracted directly from video data and contain rich visual information, such as gestures and facial expressions; the other is multimodal features obtained through contrastive learning. These features are obtained by calculating the cosine similarity between the video features and the cosine similarity of each gloss in the gloss vocabulary. They contain semantic information and help the model better understand the video content. Before fusion, an adaptive weight generator is used to generate adaptive weights, thereby achieving effective fusion and obtaining... : ; ; ; ; The above formula is a normalization process, where This is a placeholder for any real number, used only for formal definition. Numerical values ​​representing the time dimension. , For variance, It is a very small value; It is a scaling factor. It is a bias term.

[0035] ; This formula is for projection without bias. It is a learnable weight matrix.

[0036] ; This is the activation function.

[0037] ; ; ; The Generate() function allows for flexible weighting, enabling... and Effective fusion is performed, and finally, feature transformation is carried out through Fusion Adapter FA(·) to obtain the recognition result. : ; Where N represents the number of predicted glosses, m represents the number of gloss categories plus 1; Result is a probability matrix, which can be decoded by CTCDecoder to obtain the predicted glosses.

[0038] In this embodiment of the invention, high-level abstract features and low-level temporal semantics can be extracted from sign language videos, and more comprehensive multimodal fusion features can be formed through contrastive learning and dynamic weight fusion modules, thereby improving the accuracy of continuous sign language recognition.

[0039] Example 2, please refer to Figure 2 , Figure 2 This is a schematic diagram of the structural composition of the multimodal continuous sign language recognition device based on contrastive learning in an embodiment of the present invention.

[0040] like Figure 2 As shown, a multimodal continuous sign language recognition device based on contrastive learning is provided, the device comprising: Feature enhancement module 201: used to input multimodal data into the modal feature enhancement module in the sign language recognition model for feature extraction and enhancement processing to obtain enhanced feature data. The sign language recognition model includes a modal feature enhancement module, a contrast learning module, and a feature fusion module. The multimodal data includes text-annotated data and continuous sign language video data. In a specific implementation of this invention, the step of inputting multimodal data into the modal feature enhancement module of the sign language recognition model for feature extraction processing to obtain enhanced feature data includes: after inputting the multimodal data into the sign language recognition model, the modal feature enhancement module calls the corresponding data encoder to perform feature extraction processing according to different modal data in the multimodal data to obtain first extracted feature data and second extracted feature data, wherein the first extracted feature data is feature data corresponding to text annotation data, and the second extracted feature data is feature data corresponding to continuous sign language video data; the first extracted feature data and the second extracted feature data are input into the Adapter unit of the modal feature enhancement module for feature processing to form first processed feature data and second processed feature data, wherein the Adapter unit consists of two linear layers and a layer normalization, and a layer normalization and a ReLU activation function are nested between the two linear layers; the first processed feature data and the second processed feature data are input into the dynamic stacked cross-attention unit of the modal feature enhancement module for feature depth interaction and information fusion enhancement processing to form enhanced feature data.

[0041] Furthermore, the step of calling the corresponding data encoder to perform feature extraction processing based on different modal data in the multimodal data in the modal feature enhancement module to obtain first extracted feature data and second extracted feature data includes: calling the gloss data encoder to perform feature extraction processing based on the text annotation data in the multimodal data in the modal feature enhancement module to obtain first extracted feature data, wherein the gloss data encoder is a pre-trained multilingual bidirectional encoder used to extract sign language vocabulary feature data; and calling the video data encoder to perform feature extraction processing based on the continuous sign language video data in the multimodal data in the modal feature enhancement module to obtain second extracted feature data, wherein the video data encoder is a pre-trained SlowFastSign network used to extract visual feature data related to sign language.

[0042] Furthermore, the step of inputting the first extracted feature data and the second extracted feature data into the Adapter unit of the modal feature enhancement module for feature processing to form the first processed feature data and the second processed feature data includes: inputting the first extracted feature data into the Gloss2Gloss Adapter unit of the modal feature enhancement module for feature processing to form the first processed feature data; and inputting the second extracted feature data into the Video2Video Adapter unit of the modal feature enhancement module for feature processing to form the second processed feature data.

[0043] Furthermore, the step of inputting the first processed feature data and the second processed feature data into the dynamic stacked cross-attention unit in the modal feature enhancement module for feature depth interaction and information fusion enhancement processing to form enhanced feature data includes: after inputting the first processed feature data and the second processed feature data into the dynamic stacked cross-attention unit in the modal feature enhancement module, the dynamic stacked cross-attention unit uses the second processed feature data as a key K and a value V, and the first processed feature data as a query Q; and iteratively processes the query Q (the first processed feature data) and the key K and value V (the second processed feature data) in the MLP layer and residual connection within the dynamic stacked cross-attention unit to form enhanced feature data.

[0044] Specifically, such as Figure 4 As shown, the framework for multimodal continuous sign language recognition mainly includes Multimodal Feature Enhancement (MFE), Adaptive Contrastive Similarity (ACS), and Dynamic Weight Allocation (DWA).

[0045] Multimodal data is input into the modality feature enhancement module of the sign language recognition model for feature extraction and enhancement. The multimodal data includes text-annotated data (gloss data) and continuous sign language video data (video data). The feature extraction from gloss data and video data differs significantly. Gloss modality primarily focuses on the semantic information of the language, while video modality needs to capture visual information such as hand movements and facial expressions. Given a sign language video V = {v1, v2, ..., v...} with n frames... nGiven a vocabulary G={g1,g2,...,gm} consisting of m-1 glosses and 1 blank, the goal is to predict consecutive handwritten sentences with a glosses in a video using a designed neural network Net(·). Result= {r1,r2,...,r a} The size of m is equal to the number of categories in the sign language dataset gloss plus 1. The addition of 1 is because blank is also included in this collection. ; Gloss Data Encoder: To enhance the semantic richness of text features, a pre-trained multilingual bidirectional encoder (LaBSE, Language-agnostic BERT Sentence Embeddings) is used to extract sign language vocabulary (gloss) features. Specifically, six custom sentence templates are constructed, and each gloss is extracted from the gloss vocabulary and embedded into these six template sentences, generating a set of sentences related to that gloss. These sentences are then input into LaBSE for feature extraction, outputting six feature vectors. These six feature vectors are averaged to obtain a feature vector that accurately reflects the comprehensive semantics of the gloss. These feature representations not only accurately capture the core semantic information of each gloss but also further enrich the semantic connotation of the features through the incorporation of contextual information. The gloss data encoder E... g (·) Extract features for each g in vocabulary G: ; Where F g It is the result of extracting features from the gloss vocabulary through the gloss data encoder. , where 768 is the dimension of the hidden layer.

[0046] Video Data Encoder: A pre-trained SlowFastSign network is used as the video feature encoder to extract visual features related to sign language. Specifically, a single sign language video is input, processed through SlowFastSign's bidirectional feature fusion module and path feature enhancement module, to obtain the visual feature output. It's important to note that only the feature enhancement part of the path feature enhancement module is used, while its classifier layer is removed, to ensure that the video encoder's output can be directly used for subsequent feature fusion and contrastive learning tasks. This design fully utilizes the SlowFastSign architecture to capture the spatiotemporal information of the video, providing high-quality visual feature representations for continuous sign language recognition tasks. Given a video encoder... : ; in It is the result of extracting features from sign language videos through a video encoder. , where 1024 represents the dimension of the hidden layer.

[0047] The modal feature enhancement module contains Adapter units, which are used to improve the diversity of feature samples. Each Adapter unit consists of two linear layers and a layer normalization function (LayerNorm), with a layer normalization function and a ReLU activation function nested between the two linear layers. The Adapter units used all have the same architecture and are additionally named Gloss2Gloss (G2G) Adapter, Video2Video Adapter, Similarity Adapter, and FusionAdapter. Given features... Where D represents the hidden layer dimension, through linear layers Layer normalization function ReLU activation function and residuals Where 1024 is the dimension of the hidden layer: ; To effectively integrate features from both text and video modalities, a Modality Feature Enhancement (MFE) module is proposed. This module extracts gloss and video features separately using two dedicated encoders and ensures the consistency and complementarity of the two modal features through an adaptive adjustment mechanism. In addition, a Dynamic Stacked Cross-Attention (DSCA) module is designed to further enhance the semantic representation of the gloss features by using the cross-attention mechanism as the core.

[0048] In the modal feature enhancement process, the Gloss2Gloss (G2G) Adapter and Video2Video (V2V) Adapter are introduced to align gloss features with video features. This adaptive module adjusts the feature scale to ensure that gloss and video features maintain dimensionality consistency, thus providing more consistent input for subsequent feature fusion. Furthermore, the adaptive module fine-tunes the outputs of the gloss encoder and video encoder, further optimizing feature representation and improving the model's ability to integrate features from different modalities. The gloss features output by the gloss encoder... The calibrated gloss features are obtained after gloss2gloss Adapter G2G(·). Meanwhile, the video features output by the video encoder The calibrated video features are obtained after Video2VideoAdapter V2V(·). : ; ; like Figure 5 The diagram shows the framework of the Dynamic Stacked Cross-Attention (DSCA) unit. DSCA is proposed to achieve deep interaction and information fusion between gloss features and video features, thereby enriching the gloss features. This unit uses video features F... video As keys and values, gloss features As a query, the input is passed to the cross-attention mechanism, then through the MLP layer and residual connections. The residual output serves as the query, while video features remain as keys and values. Further semantic information is extracted through cross-attention, MLP, and residual connections. After N iterations, the iterative output is used by an α generator to generate flexible weight values. Finally, the original input gloss features and the iterative output are effectively fused, allowing the gloss features to extract rich contextual information from the video features and enhancing their semantic representation. The MLP layer consists of two linear layers with a QuickGELU activation function embedded between them. The iteration number M in the dynamically stacked cross-attention is set to 2. Given the features output by Gloss2GlossAdapter... Features of Video2VideoAdapter output Enhanced feature data can be obtained through dynamic stacking of cross attention. : ; ; ; ; ; in (·) represents the layer normalization function. (·) denotes the normalized exponential function. (·) indicates projection mapping. (·) is one of the activation functions. (·) is the weight generator.

[0049] Contrast learning module 202: used to input the enhanced feature data into the contrast learning module in the sign language recognition model for contrast learning processing to obtain contrast learning results; In a specific implementation of this invention, the step of inputting the enhanced feature data into the contrast learning module of the sign language recognition model for contrast learning processing to obtain a contrast learning result includes: after inputting the enhanced feature data into the contrast learning module of the sign language recognition model, the contrast learning module uses the enhanced feature data and the second processed feature data to perform cosine similarity calculation processing to obtain a cosine similarity calculation result; and then, within the similarity adapter of the contrast learning module, the cosine similarity calculation result is used for comparison processing to obtain a contrast learning result.

[0050] Specifically, through modal feature enhancement, not only were semantically rich gloss features obtained (significantly enhanced by video features), but also accurate video features were extracted. Subsequently, the features of these two modalities were normalized, and their cosine similarity was calculated. This process allows us to determine which gloss in the gloss vocabulary is closest to the feature in each temporal dimension of the video, thus providing accurate semantic mapping for continuous sign language recognition. During experiments, it was found that relying solely on cosine similarity as the output resulted in very poor model training results. Therefore, a similarity adapter module was introduced. This module, structurally similar to other adaptive modules and with a simple architecture, significantly improves model performance, allowing gloss features and video features to play a better role at the similarity level. Given video features… and semantically rich gloss features ,calculate and Cosine similarity between And through the Similarity Adapter module Ideal comparison results can be obtained. : ; ; in This represents the magnitude of the eigenvector.

[0051] Fusion prediction module 203: is used to input the contrast learning results into the feature fusion module in the sign language recognition model, and dynamically assign weights to different features based on the weight generator and perform fusion processing in the feature fusion module to output the sign language recognition prediction result.

[0052] In a specific implementation of this invention, the step of dynamically assigning weights to different features and performing fusion processing based on a weight generator in the feature fusion module to output sign language recognition prediction results includes: the feature fusion module dynamically assigning corresponding weight parameters to different features using a weight generator; the feature fusion module using the corresponding weight parameters assigned to different features to perform feature fusion processing with the contrast learning results and the second extracted feature data to form fused feature data; and the feature fusion module using the Fusion Adapter FA(·) function to perform feature transformation processing on the fused feature data to form sign language recognition prediction results and output them.

[0053] Specifically, the Dynamic Weight Allocation (DWA) module performs temporal residual fusion on the spatial embeddings of multimodal sign language to facilitate the network's extraction of higher-order abstract features while avoiding the vanishing gradient problem. The core function of DWA is to dynamically assign weights to different features through a weight generator, thereby optimizing the feature fusion process and ultimately outputting classification results. The weight generator can dynamically adjust the relative importance between different features, thus promoting effective feature fusion. The weight generator consists of two linear layers, nested with a LayerNorm and a ReLU activation function. After processing by these layers, the output is further processed by a sigmoid activation function to generate flexible weight values. These weights can be dynamically adjusted according to the characteristics of the input features, ensuring the efficiency and adaptability of the feature fusion process.

[0054] Specifically, one feature is the feature from the output of the video encoder. These features are extracted directly from video data and contain rich visual information, such as gestures and facial expressions; the other is multimodal features obtained through contrastive learning. These features are obtained by calculating the cosine similarity between the video features and the various glosses in the gloss vocabulary. They contain semantic information and help the model better understand the video content. Before fusion, an adaptive weight generator is used to generate weights, thus achieving effective fusion. : ; ; ; ; The above formula is a normalization process, where This is a placeholder for any real number, used only for formal definition. Numerical values ​​representing the time dimension. , For variance, It is a very small value; It is a scaling factor. It is a bias term.

[0055] ; This formula is for projection without bias. It is a learnable weight matrix.

[0056] ; This is the activation function.

[0057] ; ; ; The Generate() function allows for flexible weighting, enabling... and Effective fusion is performed, and finally, feature transformation is carried out through Fusion Adapter FA(·) to obtain the recognition result. : ; Where N represents the number of predicted glosses, m represents the number of gloss categories plus 1; Result is a probability matrix, which can be decoded by CTCDecoder to obtain the predicted glosses.

[0058] In this embodiment of the invention, high-level abstract features and low-level temporal semantics can be extracted from sign language videos, and more comprehensive multimodal fusion features can be formed through contrastive learning and dynamic weight fusion modules, thereby improving the accuracy of continuous sign language recognition.

[0059] This invention provides a computer-readable storage medium storing a computer program. When executed by a processor, this program implements the multimodal continuous sign language recognition method of any of the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium that stores or transmits information in a readable form by a device (e.g., a computer, a mobile phone), and can be a read-only memory, a disk, or an optical disk, etc.

[0060] This invention also provides a computer application running on a computer, which is used to execute the multimodal continuous sign language recognition method of any of the above embodiments.

[0061] also, Figure 3 This is a schematic diagram of the structural composition of the electronic device in an embodiment of the present invention.

[0062] This invention also provides an electronic device, such as... Figure 3 As shown. The electronic device includes a processor 302, a memory 303, an input unit 304, and a display unit 305, among other devices. Those skilled in the art will understand that... Figure 3 The structural components of the illustrated electronic device do not constitute a limitation on all devices and may include more or fewer components than illustrated, or combine certain components. Memory 303 can be used to store application program 301 and various functional modules. Processor 302 runs application program 301 stored in memory 303, thereby performing various functional applications and data processing of the device. Memory can be internal memory or external memory, or both. Internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. External memory may include hard disks, floppy disks, ZIP disks, USB flash drives, magnetic tapes, etc. The memory disclosed in this invention includes, but is not limited to, these types of memory. The memory disclosed in this invention is only an example and not a limitation.

[0063] Input unit 304 is used to receive signal input and user-input keywords. Input unit 304 may include a touch panel and other input devices. The touch panel can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive the corresponding connection device according to a pre-set program; other input devices may include, but are not limited to, one or more of physical keyboards, function keys (such as play control buttons, power buttons, etc.), trackballs, mice, joysticks, etc. Display unit 305 can be used to display user-input information or information provided to the user, as well as various menus of the terminal device. Display unit 305 may be in the form of a liquid crystal display, organic light-emitting diode, etc. Processor 302 is the control center of the terminal device, connecting various parts of the entire device through various interfaces and lines, and performing various functions and processing data by running or executing software programs and / or modules stored in memory 303, and calling data stored in memory.

[0064] As one embodiment, the electronic device includes: one or more processors 302, a memory 303, and one or more application programs 301, wherein the one or more application programs 301 are stored in the memory 303 and configured to be executed by the one or more processors 302, and the one or more application programs 301 are configured to execute the multimodal continuous sign language recognition method corresponding to any of the embodiments described above.

[0065] In this embodiment of the invention, high-level abstract features and low-level temporal semantics can be extracted from sign language videos, and more comprehensive multimodal fusion features can be formed through contrastive learning and dynamic weight fusion modules, thereby improving the accuracy of continuous sign language recognition.

[0066] Furthermore, the above provides a detailed description of a multimodal continuous sign language recognition method and related apparatus based on contrastive learning provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal continuous sign language recognition method based on contrastive learning, characterized in that, The method includes: Multimodal data is input into the modal feature enhancement module in the sign language recognition model for feature extraction and enhancement processing to obtain enhanced feature data. The sign language recognition model includes a modal feature enhancement module, a contrast learning module, and a feature fusion module. The multimodal data includes text-annotated data and continuous sign language video data. The enhanced feature data is input into the contrastive learning module in the sign language recognition model for contrastive learning processing to obtain the contrastive learning result. The contrastive learning results are input into the feature fusion module of the sign language recognition model. In the feature fusion module, weights are dynamically assigned to different features based on the weight generator and fusion processing is performed to output the sign language recognition prediction result.

2. The multimodal continuous sign language recognition method according to claim 1, characterized in that, The process of inputting multimodal data into the modal feature enhancement module of the sign language recognition model for feature extraction to obtain enhanced feature data includes: After the multimodal data is input into the sign language recognition model, the modal feature enhancement module calls the corresponding data encoder to perform feature extraction processing according to the different modal data in the multimodal data, and obtains the first extracted feature data and the second extracted feature data, wherein the first extracted feature data is the feature data corresponding to the text annotation data, and the second extracted feature data is the feature data corresponding to the continuous sign language video data. The first extracted feature data and the second extracted feature data are input into the Adapter unit in the modal feature enhancement module for feature processing to form the first processed feature data and the second processed feature data. The Adapter unit consists of two linear layers and a layer normalization, and a layer normalization and a ReLU activation function are nested between the two linear layers. The first processed feature data and the second processed feature data are input into the dynamic stacked cross-attention unit in the modal feature enhancement module for enhanced processing of feature depth interaction and information fusion to form enhanced feature data.

3. The multimodal continuous sign language recognition method according to claim 2, characterized in that, The modal feature enhancement module calls the corresponding data encoder to perform feature extraction processing based on different modal data in the multimodal data to obtain first extracted feature data and second extracted feature data, including: In the modal feature enhancement module, the gloss data encoder is called to perform feature extraction processing based on the text annotation data in the multimodal data to obtain the first extracted feature data. The gloss data encoder is a pre-trained multilingual bidirectional encoder used to extract sign language vocabulary feature data. In the modal feature enhancement module, the video data encoder is invoked to perform feature extraction processing based on the continuous sign language video data in the multimodal data to obtain the second extracted feature data. The video data encoder is a pre-trained SlowFastSign network used to extract visual feature data related to sign language.

4. The multimodal continuous sign language recognition method according to claim 2, characterized in that, The step of inputting the first extracted feature data and the second extracted feature data into the Adapter unit of the modal feature enhancement module for feature processing to form first processed feature data and second processed feature data includes: The first extracted feature data is input into the Gloss2Gloss Adapter unit in the modal feature enhancement module for feature processing to form the first processed feature data; The second extracted feature data is input into the Video2Video Adapter unit in the modal feature enhancement module for feature processing to form the second processed feature data.

5. The multimodal continuous sign language recognition method according to claim 2, characterized in that, The step of inputting the first processed feature data and the second processed feature data into the dynamic stacked cross-attention unit in the modal feature enhancement module for feature depth interaction and information fusion enhancement processing to form enhanced feature data includes: After the first processed feature data and the second processed feature data are input into the dynamic stacked cross attention unit in the modal feature enhancement module, the dynamic stacked cross attention unit uses the second processed feature data as the key K and the value V, and the first processed feature data as the query Q. Enhanced feature data is formed by iteratively processing the first processed feature data as query Q and the second processed feature data as key K and value V in the MLP layer and residual connection within the dynamically stacked cross-attention unit.

6. The multimodal continuous sign language recognition method according to claim 1, characterized in that, The step of inputting the enhanced feature data into the contrastive learning module of the sign language recognition model for contrastive learning processing to obtain the contrastive learning result includes: After the enhanced feature data is input into the contrast learning module in the sign language recognition model, the contrast learning module uses the enhanced feature data and the second processed feature data to perform cosine similarity calculation to obtain the cosine similarity calculation result. The cosine similarity calculation result is used for comparison processing within the similarity adapter of the comparison learning module to obtain the comparison learning result.

7. The multimodal continuous sign language recognition method according to claim 1, characterized in that, The feature fusion module dynamically assigns weights to different features based on a weight generator and performs fusion processing to output sign language recognition prediction results, including: The feature fusion module uses a weight generator to dynamically assign corresponding weight parameters to different features; The feature fusion module uses the weight parameters corresponding to different feature assignments to perform feature fusion processing on the comparative learning results and the second extracted feature data to form fused feature data. The feature fusion module uses the Fusion Adapter FA(·) function to perform feature transformation processing on the fused feature data, forming a sign language recognition prediction result and outputting it.

8. A multimodal continuous sign language recognition device based on contrastive learning, characterized in that, The device includes: Feature enhancement module: This module is used to input multimodal data into the modal feature enhancement module of the sign language recognition model for feature extraction and enhancement processing to obtain enhanced feature data. The sign language recognition model includes a modal feature enhancement module, a contrastive learning module, and a feature fusion module. The multimodal data includes text-annotated data and continuous sign language video data. Contrast learning module: used to input the enhanced feature data into the contrast learning module in the sign language recognition model for contrast learning processing to obtain contrast learning results; Fusion prediction module: used to input the contrast learning results into the feature fusion module in the sign language recognition model, and dynamically assign weights to different features based on the weight generator and perform fusion processing in the feature fusion module to output the sign language recognition prediction result.

9. An electronic device comprising a processor and a memory, characterized in that, The processor runs a computer program or code stored in the memory to implement the multimodal continuous sign language recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium for storing computer programs or code, characterized in that, When the computer program or code is executed by a processor, the multimodal continuous sign language recognition method as described in any one of claims 1 to 7 is implemented.

Images