A sign language recognition method based on cross-modal continuous timing constraints

Abstract

The application discloses a sign language recognition method based on cross-modal continuous time sequence constraints. In the training of a sign language visual recognition network model, an auxiliary language model and a cross-modal continuous time sequence constraint module are introduced. Video samples are input into the sign language visual recognition network model to obtain visual features, and then corresponding visual intermediate features are obtained through multi-layer perceptron projection. Auxiliary language sequences are input into the auxiliary language model to obtain language features, and then corresponding language intermediate features are obtained through multi-layer perceptron projection. Then, the visual intermediate features and the language intermediate features are input into the cross-modal continuous time sequence constraint module to calculate the inter-modal output difference index and the cross-modal continuous constraint loss, and the training of the sign language visual recognition network model is completed. The sign language recognition is performed through the trained sign language visual recognition network model. The application enhances the performance of the algorithm, thereby promoting the application of artificial intelligence in the fields of action recognition and sign language recognition.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to a sign language recognition method based on cross-modal continuous temporal constraints. Background Technology

[0002] Sign language is a naturally evolved, multi-channel, and geographically dispersed language, encompassing rich gestures, facial expressions, and body postures. The Continuous Sign Language Recognition (CSLR) task aims to detect individual sign language words (glosses) from continuous sign language videos, which is crucial for improving the quality of life in deaf communities. Existing CSLR methods typically rely on weakly supervised mechanisms, such as Connectionist Temporal Classification (CTC), to align visual input sequences with sequentially arranged sign language words without detailed labels.

[0003] Continuous action recognition technology has been extensively and deeply studied in recent years. However, existing methods still have certain limitations in capturing linguistic semantics and fusing visual features. Current technologies often struggle to cope with the diverse forms of sign language expression and complex background interference. First, sign language expression is highly diverse and variable; differences between sign language users, as well as variations in the speed and force of gestures, all affect the recognition results. Second, complex background environments and diverse lighting conditions can also interfere with visual feature extraction, making it difficult for models to accurately capture sign language movements. Furthermore, existing methods also perform poorly in handling occlusion and partial viewpoint loss; when gestures are partially occluded or the viewpoint changes, the model's recognition accuracy often drops significantly.

[0004] Existing methods primarily focus on feature extraction and fusion, but by neglecting the role of linguistic semantics, they often lead to false positives and overfitting in the recognition results. In sign language recognition, how to better capture and fuse features from both visual and linguistic modalities to improve recognition accuracy remains a pressing challenge. Current technologies mainly rely on weakly supervised methods, such as CTC loss, to align visual input sequences with sign language vocabulary lists, but these methods still have shortcomings in semantic alignment and modality fusion. Summary of the Invention

[0005] The purpose of this application is to provide a sign language recognition method based on cross-modal continuous temporal constraints. In view of the problems of lack of data and difficulty in training current data-driven sequence recognition methods, this application proposes a cross-modal constraint method by constructing cross-modal equivalent data, which aims to improve the recognition accuracy and generalization ability of the recognition model.

[0006] To achieve the above objectives, the technical solution of this application is as follows:

[0007] A sign language recognition method based on cross-modal continuous temporal constraints includes:

[0008] The sign language visual recognition network model is trained by introducing an auxiliary language model and a cross-modal continuous temporal constraint module during training, including:

[0009] Initialize the network parameters of the sign language visual recognition network model and the auxiliary language model;

[0010] Video samples are input into a sign language visual recognition network model to obtain visual features, and then the corresponding intermediate visual features are obtained by projection through a multilayer perceptron.

[0011] Construct an auxiliary language sequence corresponding to the video sample, input it into the auxiliary language model to obtain language features, and then obtain the corresponding intermediate language features through multilayer perceptron projection;

[0012] Visual intermediate features and linguistic intermediate features are input into the cross-modal continuous temporal constraint module to calculate the inter-modal output difference index and the cross-modal continuous constraint loss;

[0013] The training of the sign language visual recognition network model was completed by minimizing the cross-modal continuous constraint loss;

[0014] The video to be recognized is input into the trained sign language visual recognition network model to obtain the recognition result.

[0015] Furthermore, the sign language visual recognition network model includes a feature extraction neural network layer for extracting visual features and a classification module for classifying the visual features.

[0016] Furthermore, the visual features are the visual features extracted by any layer in the feature extraction neural network.

[0017] Furthermore, the auxiliary language model includes an encoder and a decoder.

[0018] Furthermore, the language features are the language features extracted from any layer of the encoder.

[0019] Furthermore, the intermodal output difference index is calculated using the following normalized dot product similarity function:

[0020]

[0021] in, This represents the natural exponential function. This represents the learnable scaling factor. Indicates the first Visual intermediate features corresponding to frame video Indicates the first Auxiliary intermediate variables corresponding to each auxiliary sequence state Indicates the first Auxiliary intermediate variables corresponding to each auxiliary sequence state The similarity is calculated, and K represents the number of states in the auxiliary language sequence.

[0022] Furthermore, the cross-modal continuous constraint loss is calculated using the CTC loss function:

[0023]

[0024] in, This represents the cross-modal continuous constraint loss. Let S represent the CTC loss function, and S represent the intermodal output difference index matrix. The function generates values ​​from 0 to 1. sequence of integers.

[0025] The sign language recognition method based on cross-modal continuous temporal constraints proposed in this application has the following advantages compared with the prior art:

[0026] This significantly reduces the time required for training convergence, accelerating the network training process and thus saving computational overhead and energy consumption. Experiments show that the technical solution presented in this application can improve network performance and generalization ability without increasing the number of additional parameters or the computational cost of the final model, i.e., it improves parameter efficiency. Furthermore, the network trained in this application produces outputs that are more stable and better match the key features of the input data compared to traditional methods, exhibiting better interpretability. This technical solution provides effective and reliable supervision information for recognition algorithms, enhancing their performance and thus promoting the application of artificial intelligence in fields such as action recognition and sign language recognition. Attached Figure Description

[0027] Figure 1 This is a flowchart of the sign language recognition method based on cross-modal continuous temporal constraints proposed in this application.

[0028] Figure 2 This is a structural diagram of a specific embodiment of the sign language visual recognition network model of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] In one embodiment, such as Figure 1As shown, a sign language recognition method based on cross-modal continuous temporal constraints is proposed, including:

[0031] Step S1: Train the sign language visual recognition network model, and introduce an auxiliary language model and a cross-modal continuous temporal constraint module during training.

[0032] In this embodiment, an auxiliary language model and a cross-modal continuous temporal constraint module are introduced to train the sign language visual recognition network model. The purpose of these modules is to train the sign language visual recognition network model and to optimize and improve the accuracy and word error rate of existing recognition models by constructing cross-modal continuous temporal constraints.

[0033] The sign language visual recognition network model constructed in this embodiment is the main model, which generally includes a feature extraction neural network layer for extracting visual features and a classification module for classifying visual features.

[0034] In a specific embodiment, such as Figure 2 As shown, the main model includes a backbone network (e.g., a convolutional neural network (CNN)) for processing image input, a temporal convolutional module for temporal fusion, a recurrent neural network (RNN, e.g., an LSTM network) for language modeling, and a classification module for obtaining prediction results. The classification module can employ a multilayer perceptron (MLP) and a softmax activation function. The backbone network, temporal convolutional module, and recurrent neural network all belong to the feature extraction neural network layers used to extract visual features. This application does not limit the specific structure of the main model; it can also be various recognition models with other structures, which will not be elaborated further below.

[0035] In a typical visual recognition network model, visual features are obtained through various network layers before the classification module. The classification module then uses these visual features to obtain a prediction result, which is the recognition result obtained by the recognition model.

[0036] Generally, in order to perform visual recognition, a visual recognition network model is first built and trained, and then the video to be recognized is input into the trained model to obtain the final recognition result.

[0037] To better train the visual recognition network model, this application introduces an auxiliary language model and a cross-modal continuous temporal constraint module during training.

[0038] The introduced auxiliary language model includes an encoder for processing language input and a decoder for outputting language for the task. The auxiliary language model needs to be pre-trained before it can be used to assist in training the main model. When pre-training the auxiliary language model, it can be pre-trained on a large text dataset through self-supervision or other methods, or the weights can be pre-trained using various open-source network models.

[0039] The introduced cross-modal continuous temporal constraint module includes a modal output difference index calculation unit and a cross-modal continuous constraint loss function calculation unit.

[0040] When training the sign language visual recognition network model, the following steps are performed:

[0041] Step S101: Initialize the network parameters of the sign language visual recognition network model and the auxiliary language model.

[0042] Specifically, the backbone network of the main model and the network parameters of the auxiliary language model can be loaded with pre-trained network parameters. The remaining parameters of the sign language visual recognition network model, i.e., the main model, can be initialized using normal random distribution, Xavier, He, or other methods.

[0043] Step S102: Input the video sample into the sign language visual recognition network model to obtain visual features, and then obtain the corresponding visual intermediate features through multilayer perceptron projection.

[0044] In this technical field, visual features are obtained by extracting features from videos through various network layers. In this embodiment, different network layer outputs can be selected as visual features in the main model as appropriate. Typically, the output of the temporal convolutional module used for temporal fusion can be selected as the visual features. Alternatively, the output of a recurrent neural network can be selected as the visual feature, or the visual feature extracted from any layer of a feature extraction neural network.

[0045] For example, video samples Input to the main model In the process, video features are obtained. The video features are projected onto a common high-dimensional constrained space using a multilayer perceptron (MLP) to obtain intermediate visual features. Used for cross-modal continuous temporal constraint calculation. This feature dimension is typically [value missing]. , This represents the number of samples in each batch during the mini-batch training process. Input the number of video frames (or the number of downsampled video frames). Let be the dimension of the feature vectors constraining the space. To complete mini-batch training, Dimensions may need to be padded to ensure that the total sequence length of each sample is the same.

[0046] Among them, video features Visual intermediate features are obtained by projecting them onto a common high-dimensional constrained space using a multilayer perceptron (MLP). , means as follows:

[0047] .

[0048] Step S103: Construct an auxiliary language sequence corresponding to the video sample, input it into the auxiliary language model, obtain language features, and then obtain the corresponding intermediate language features through multilayer perceptron projection.

[0049] This embodiment constructs auxiliary language sequences corresponding to video samples. The equivalent sequences are constructed from the videos in the training dataset. Auxiliary language sequences are descriptions of the input videos using natural text and contain the information needed for the recognition task. Taking sign language recognition as an example, irrelevant information such as clothing and background in the video is irrelevant to sign language recognition, but descriptions of the characters' actions, emotions, and expressions are relevant to the recognition task. If the equivalent auxiliary language sequences... If it contains information relevant to the recognition task, it can be considered an equivalent sequence. Equivalent auxiliary language sequence Its equivalent length The number of target (Gloss) sign language words should be consistent with the number of target (GroundTruth) sign language words in the training samples. The equivalent length refers to the number of paragraphs, also known as the number of states. Each paragraph describes one target sign language word, and the order of the paragraphs is consistent with the order in which the sign language words appear.

[0050] Constructed auxiliary language sequence Input to auxiliary language model In the process, language features are obtained. This feature is obtained by projecting it onto a common high-dimensional constrained space using a multilayer perceptron (MLP) to obtain intermediate language features. Used for cross-modal continuous temporal constraint calculation. This feature dimension is typically [value missing]. , This represents the number of samples in each batch during the mini-batch training process. The number of states in the auxiliary language sequence (which is also equal to the number of sign language words in the target sequence). Let be the dimension of the feature vectors constraining the space. To complete mini-batch training, Dimensions may need to be padded to ensure that the total sequence length of each sample is the same.

[0051] Among them, language features Language intermediate features are obtained by projecting them onto a common high-dimensional constrained space using a multilayer perceptron (MLP). , means as follows:

[0052] .

[0053] Multilayer perceptron (MLP) is a relatively mature technology in this field, and will not be elaborated on here.

[0054] It should be noted that the language features can be selected from the output of the encoder or from the output of any layer of the encoder; this application does not impose any restrictions on this.

[0055] Step S104: Input the visual intermediate features and language intermediate features into the cross-modal continuous temporal constraint module, and calculate the inter-modal output difference index and cross-modal continuous constraint loss.

[0056] During the training process of each batch, the visual intermediate features obtained from the above steps are... and intermediate features of language Input to the cross-modal continuous timing constraint module.

[0057] In a specific embodiment, the output difference index between modalities is calculated using the following normalized dot product similarity function:

[0058]

[0059] in, This represents the natural exponential function. This represents the learnable scaling factor. Indicates the first Visual intermediate features corresponding to frame video Indicates the first Auxiliary intermediate variables corresponding to each auxiliary sequence state The calculated similarity is represented by K, where K represents the number of states in the auxiliary language sequence. This yields the intermodal output difference index matrix. . Indicates the first Auxiliary intermediate variables corresponding to each auxiliary sequence state.

[0060] In one specific embodiment, the cross-modal continuous constraint loss is calculated using the CTC loss function:

[0061]

[0062] in, This represents the cross-modal continuous constraint loss. Let S represent the CTC loss function, and S represent the intermodal output difference index matrix. The Connectionist Temporal Classification (CTC) loss function is a relatively mature technique in this field, and will not be elaborated upon here. The function generates values ​​from 0 to 1. The integer sequence is used as a pseudo-label for the CTC function.

[0063] Step S105: Complete the training of the sign language visual recognition network model by minimizing the cross-modal continuous constraint loss.

[0064] This step trains the sign language visual recognition network model by minimizing the cross-modal continuous constraint loss. The network parameters of the main model are then saved for subsequent recognition use.

[0065] Step S2: Input the video to be recognized into the trained sign language visual recognition network model to obtain the recognition result.

[0066] After training, the main model parameters are loaded, and the trained sign language visual recognition network model is used to perform sign language recognition. The video sequence to be recognized is then used. Input the main model to obtain the prediction results. During the recognition process, the auxiliary language model and the cross-modal continuous temporal constraint module will not participate in the recognition, which will not be elaborated here.

[0067] Experiments were conducted on the CSL-Daily and Phoenix-2014 datasets to illustrate the method proposed in this application. The results were compared with existing methods such as BN-TIN, FCN, VAC, CTCA, SEN_CSLR, and CorrNET. The experimental results are shown in Table 1.

[0068] Table 1

[0069]

[0070] Experimental data show that the word error rate (WER) of the method in this application is lower than that of the existing technology and has better recognition performance.

[0071] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A sign language recognition method based on cross-modal continuous temporal constraints, characterized in that, The sign language recognition method based on cross-modal continuous temporal constraints includes: The sign language visual recognition network model is trained by introducing an auxiliary language model and a cross-modal continuous temporal constraint module during training, including: Initialize the network parameters of the sign language visual recognition network model and the auxiliary language model; Video samples are input into a sign language visual recognition network model to obtain visual features, and then the corresponding intermediate visual features are obtained by projection through a multilayer perceptron. Construct an auxiliary language sequence corresponding to the video sample, input it into the auxiliary language model to obtain language features, and then obtain the corresponding intermediate language features through multilayer perceptron projection; Visual intermediate features and linguistic intermediate features are input into the cross-modal continuous temporal constraint module to calculate the inter-modal output difference index and the cross-modal continuous constraint loss; The training of the sign language visual recognition network model was completed by minimizing the cross-modal continuous constraint loss; Input the video to be recognized into the trained sign language visual recognition network model to obtain the recognition result; The intermodal output difference index is calculated using the following normalized dot product similarity function: ； in, This represents the natural exponential function. This represents the learnable scaling factor. Indicates the first Visual intermediate features corresponding to frame video Indicates the first Auxiliary intermediate variables corresponding to each auxiliary sequence state Indicates the first Auxiliary intermediate variables corresponding to each auxiliary sequence state The similarity is calculated, and K represents the number of states in the auxiliary language sequence. The cross-modal continuous constraint loss is calculated using the CTC loss function: ； in, This represents the cross-modal continuous constraint loss. Let S represent the CTC loss function, and S represent the intermodal output difference index matrix. The function generates values ​​from 0 to 1. sequence of integers.

2. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 1, characterized in that, The sign language visual recognition network model includes a feature extraction neural network layer for extracting visual features and a classification module for classifying the visual features.

3. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 2, characterized in that, The visual features are the visual features extracted by any layer in the feature extraction neural network.

4. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 1, characterized in that, The auxiliary language model includes an encoder and a decoder.

5. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 4, characterized in that, The language features are the language features extracted from any layer of the encoder.

Images