Sign language recognition method based on cross-modal continuous timing constraint

Abstract

The invention discloses a cross-modal continuous time sequence constraint-based sign language recognition method, which comprises the following steps of: introducing an auxiliary language model and a cross-modal continuous time sequence constraint module when training a sign language visual recognition network model, inputting a video sample into the sign language visual recognition network model, and obtaining visual features; a corresponding visual intermediate feature is obtained through multi-layer perceptron projection; inputting the auxiliary language sequence into an auxiliary language model to obtain language features, and then obtaining corresponding language intermediate features through multi-layer perceptron projection; and inputting the visual intermediate feature and the language intermediate feature into a cross-modal continuous time sequence constraint module, calculating an inter-modal output difference index and a cross-modal continuous constraint loss, and completing training of a sign language visual recognition network model. And sign language recognition is carried out through the trained sign language visual recognition network model. The performance of the algorithm is enhanced, so that application of artificial intelligence in the fields of action recognition, sign language recognition and the like is promoted.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a sign language recognition method based on cross-modal continuous temporal constraints. Background Art

[0002] Sign language is a naturally evolving, multi-channel, and geographically dispersed language that contains rich gestures, facial expressions, and body postures. The Continuous Sign Language Recognition (CSLR) task aims to detect individual sign language words (i.e., glosses) from continuous sign language videos, which is critical to improving the quality of life of the deaf community. Existing CSLR methods usually rely on weak supervision mechanisms, such as CTC loss (Connectionist Temporal Classification), to align visual input sequences with sequentially arranged sign language words in the absence of detailed labels.

[0003] Continuous action recognition technology has been widely and deeply studied in recent years. However, existing methods still have certain limitations in capturing language semantics and integrating visual features. Existing technologies often find it difficult to cope with the changing forms of sign language expressions and complex background interference. First of all, the expression of sign language is highly diverse and variable. The differences between different sign language users, the speed and strength of gestures will affect the recognition results. Secondly, complex background environments and diverse lighting conditions will also interfere with the extraction of visual features, making it difficult for the model to accurately capture sign language movements. In addition, existing methods also perform poorly in dealing with occlusion and partial loss of perspective. When the gesture is partially occluded or the perspective changes, the recognition accuracy of the model tends to drop significantly.

[0004] Existing methods mainly focus on feature extraction and fusion, but because they ignore the role of language semantics, they often lead to false positives and overfitting in the recognition results. In the process of sign language recognition, how to better capture and integrate the features of visual and language modalities to improve the accuracy of recognition remains a difficult problem to be solved. Existing technologies mainly rely on weak supervision methods, such as CTC loss, to align visual input sequences and sign language vocabulary lists, but these methods still have shortcomings in semantic alignment and modality fusion. Summary of the invention

[0005] The purpose of this application is to provide a sign language recognition method based on cross-modal continuous temporal constraints. In view of the problems of lack of data and difficulty in training in current data-driven sequence recognition methods, a cross-modal constraint method is proposed by constructing cross-modal equivalent data, aiming to improve the recognition accuracy and generalization ability of the recognition model.

[0006] In order to achieve the above purpose, the technical solution of this application is as follows: A sign language recognition method based on cross-modal continuous temporal constraints, comprising: Train the sign language visual recognition network model, introduce the auxiliary language model and cross-modal continuous timing constraint module during training, including: Initialize the network parameters of the sign language visual recognition network model and the auxiliary language model; The video samples are input into the sign language visual recognition network model to obtain visual features, and then the corresponding visual intermediate features are obtained through multi-layer perceptron projection; Construct an auxiliary language sequence corresponding to the video sample, input it into the auxiliary language model, obtain language features, and then project it through a multi-layer perceptron to obtain the corresponding language intermediate features; Input the visual intermediate features and the language intermediate features into the cross-modal continuous temporal constraint module to calculate the inter-modal output difference index and the cross-modal continuous constraint loss; The training of the sign language visual recognition network model is completed by minimizing the cross-modal continuous constraint loss; The video to be recognized is input into the trained sign language visual recognition network model to obtain the recognition result.

[0007] Furthermore, the sign language visual recognition network model includes a feature extraction neural network layer for extracting visual features and a classification module for classifying the visual features.

[0008] Furthermore, the visual features are visual features extracted by any layer in the feature extraction neural network layer.

[0009] Furthermore, the auxiliary language model includes an encoder and a decoder.

[0010] Furthermore, the language features are language features extracted by any layer of the encoder.

[0011] Furthermore, the output difference index between the modalities is calculated using the following normalized dot product similarity function: in, represents the natural exponential function, represents a learnable scaling factor, Indicates The visual intermediate features corresponding to the frame video, Indicates Auxiliary intermediate variables corresponding to auxiliary sequence states, Indicates Auxiliary intermediate variables corresponding to auxiliary sequence states, represents the calculated similarity, and K represents the number of states of the auxiliary language sequence.

[0012] Furthermore, the cross-modal continuous constraint loss is calculated by the CTC loss function: in, represents the cross-modal continuous constraint loss, represents the CTC loss function, S represents the output difference indicator matrix between modalities, The function generates a value from 0 to sequence of integers.

[0013] The present application proposes a sign language recognition method based on cross-modal continuous temporal constraints, which has the following beneficial effects compared with the prior art: It significantly reduces the time required for training convergence and accelerates the network training process, thereby saving computing overhead and energy consumption. Experiments have shown that the technical solution of the present application can improve network performance and generalization capabilities without increasing the number of additional parameters and the amount of calculation of the final model, that is, it improves parameter efficiency. Moreover, the output of the network trained by the present application is more stable than that of traditional methods, more consistent with the key features in the input data, and has better interpretability. The technical solution of the present application provides effective and reliable supervision information for the recognition algorithm, enhances the performance of the algorithm, and thus promotes the application of artificial intelligence in the fields of motion recognition, sign language recognition, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of the sign language recognition method based on cross-modal continuous timing constraints in this application.

[0015] Figure 2 This is a structural diagram of a specific embodiment of the sign language visual recognition network model of this application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] In one embodiment, Figure 1 As shown, a sign language recognition method based on cross-modal continuous temporal constraints is proposed, including: Step S1, training a sign language visual recognition network model, introducing an auxiliary language model and a cross-modal continuous temporal constraint module during training.

[0018] In order to train the sign language visual recognition network model, this embodiment introduces an auxiliary language model and a cross-modal continuous temporal constraint module. The purpose of the introduction is to train the sign language visual recognition network model and optimize and improve the accuracy and word error rate of the existing recognition model by constructing a cross-modal continuous time constraint.

[0019] The sign language visual recognition network model constructed in this embodiment is a main model, which generally includes a feature extraction neural network layer for extracting visual features and a classification module for classifying visual features.

[0020] In a specific embodiment, Figure 2 As shown, the main model includes a backbone network for processing image input (for example, using a convolutional neural network CNN), a time convolution module for time fusion, a recurrent neural network (RNN, for example, using an LSTM network) for language modeling, and a classification module for obtaining prediction results. The classification module can use a multi-layer perceptron (MLP) and a softmax function activation function. Among them, the backbone network, the time convolution module, and the recurrent neural network all belong to the feature extraction neural network layer for extracting visual features. This application does not limit the specific structure of the main model, and it can also be various recognition models of other structures, which will not be repeated below.

[0021] A common visual recognition network model can obtain visual features through each network layer before the classification module, and then the classification module obtains a prediction result based on the visual features, that is, the recognition result obtained by the recognition model.

[0022] Generally speaking, in order to perform visual recognition, a visual recognition network model is first built and trained, and then the video to be recognized is input into the trained model to obtain the final recognition result.

[0023] In order to better train the visual recognition network model, this application introduces an auxiliary language model and a cross-modal continuous temporal constraint module during training.

[0024] The auxiliary language model introduced includes an encoder for processing language input and a decoder for task-specific language output. The auxiliary language model needs to be pre-trained before being used to assist in training the main model. The auxiliary language model can be pre-trained on large text datasets through self-supervision or other forms, or various open source network models can be used to pre-train weights.

[0025] The introduced cross-modal continuous timing constraint module includes a modal output difference index calculation unit and a cross-modal continuous constraint loss function calculation unit.

[0026] When training a sign language visual recognition network model, perform the following steps: Step S101, initializing the network parameters of the sign language visual recognition network model and the auxiliary language model.

[0027] Specifically, the network parameters of the backbone network of the main model and the auxiliary language model can load the network parameters obtained by pre-training. The remaining parameters of the sign language visual recognition network model, that is, the main model, can be initialized using normal random distribution, Xavier, He, etc.

[0028] Step S102: input the video sample into the sign language visual recognition network model to obtain visual features, and then obtain the corresponding visual intermediate features through multi-layer perceptron projection.

[0029] In this technical field, visual features are obtained by extracting features from videos through various network layers. In this embodiment, different network layer outputs can be selected as visual features in the main model as appropriate. Usually, the output of the temporal convolution module used for temporal fusion can be selected as the visual feature. It is also possible to select the output of the recurrent neural network as the visual feature, or the visual feature extracted by any layer in the feature extraction neural network layer.

[0030] For example, the video sample Import to main model In the video, we get the video features The video features are projected into a common high-dimensional constraint space through a multi-layer perceptron MLP to obtain visual intermediate features Used for cross-modal continuous timing constraint calculation. This feature dimension is usually , is the number of samples in each batch during mini-batch training, is the number of input video frames (or the number of downsampled video frames), is the feature vector dimension of the constraint space. In order to complete the mini-batch training, The dimensions may need to be padded so that the total length of the sequence of each sample is the same.

[0031] Among them, video features The visual intermediate features are obtained by projecting them into a common high-dimensional constraint space through a multi-layer perceptron (MLP). , which is expressed as follows: .

[0032] Step S103: construct an auxiliary language sequence corresponding to the video sample, input it into the auxiliary language model, obtain language features, and then obtain the corresponding language intermediate features through multi-layer perceptron projection.

[0033] This embodiment constructs the auxiliary language sequence corresponding to the video sample , is an equivalent sequence constructed for the training dataset video. The auxiliary language sequence is a description of the input video through natural text and contains the information required for the recognition task. Taking the sign language recognition task as an example, irrelevant information such as the person’s clothing and background in the video is irrelevant to sign language recognition, but the person’s actions, emotions, expressions, etc. are related to the recognition task. If the equivalent auxiliary language sequence Contains information related to the recognition task, it can be regarded as an equivalent sequence. , its equivalent length It should be consistent with the number of target (GroundTruth) sign language words (gloss) in the training sample. The equivalent length refers to the number of paragraphs, also known as the number of states. Each paragraph is used to describe a target sign language word, and the order of the paragraphs is consistent with the order in which the sign language words appear.

[0034] Constructed auxiliary language sequence Input to auxiliary language model In the language features , this feature is projected into a common high-dimensional constraint space by a multi-layer perceptron projection MLP to obtain the language intermediate feature Used for cross-modal continuous timing constraint calculation. This feature dimension is usually , is the number of samples in each batch during mini-batch training, is the number of states in the auxiliary language sequence (which is also equal to the number of sign language words in the target sequence), is the feature vector dimension of the constraint space. In order to complete the mini-batch training, The dimensions may need to be padded so that the total length of the sequence of each sample is the same.

[0035] Among them, language features The language intermediate features are obtained by projecting them into a common high-dimensional constraint space through a multi-layer perceptron (MLP). , which is expressed as follows: .

[0036] Multi-layer perceptron MLP is a relatively mature technology in this field and will not be described in detail here.

[0037] It should be noted that the language feature can select the output of the encoder, or the output of any layer of the encoder, and this application does not impose any restrictions on this.

[0038] Step S104: input the visual intermediate features and the language intermediate features into the cross-modal continuous temporal constraint module to calculate the inter-modal output difference index and the cross-modal continuous constraint loss.

[0039] During the training of each batch, the visual intermediate features obtained in the above steps are and language intermediate features Input to the cross-modal continuous timing constraint block.

[0040] In a specific embodiment, the inter-modal output difference index is calculated using the following normalized dot product similarity function: in, represents the natural exponential function, represents a learnable scaling factor, Indicates The visual intermediate features corresponding to the frame video, Indicates Auxiliary intermediate variables corresponding to auxiliary sequence states, represents the calculated similarity, and K represents the number of states of the auxiliary language sequence. Thus, the output difference index matrix between modalities is obtained: . Indicates Auxiliary intermediate variables corresponding to auxiliary sequence states.

[0041] In a specific embodiment, the cross-modal continuous constraint loss is calculated using the CTC loss function: in, represents the cross-modal continuous constraint loss, Denotes the CTC loss function, and S denotes the inter-modal output difference indicator matrix. The Connectionist Temporal Classification (CTC) loss function is a relatively mature technology in this field and will not be described here. The function generates a value from 0 to The integer sequence of is used as the pseudo label of the CTC function.

[0042] Step S105: completing the training of the sign language visual recognition network model by minimizing the cross-modal continuous constraint loss.

[0043] This step completes the training of the sign language visual recognition network model by minimizing the cross-modal continuous constraint loss. The network parameters of the main model are saved for subsequent recognition.

[0044] Step S2: input the video to be recognized into the trained sign language visual recognition network model to obtain a recognition result.

[0045] After the training is completed, the main model parameters are loaded, and the sign language recognition is performed through the trained sign language visual recognition network model. Input the main model and get the prediction results During the recognition process, the auxiliary language model and the cross-modal continuous timing constraint module will not participate in the recognition and will not be described here.

[0046] For the method in this application, experiments were conducted on the CSL-Daily and Phoenix-2014 datasets, and compared with the existing technical methods BN-TIN, FCN, VAC, CTCA, SEN_CSLR and CorrNET. The experimental results are shown in Table 1: Table 1 Experimental data show that the word error rate (WER) of the method of the present application is lower than that of the existing technical solutions and has better recognition effect.

[0047] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A sign language recognition method based on cross-modal continuous temporal constraints, characterized in that: The sign language recognition method based on cross-modal continuous temporal constraints includes: Train the sign language visual recognition network model, introduce the auxiliary language model and cross-modal continuous timing constraint module during training, including: Initialize the network parameters of the sign language visual recognition network model and the auxiliary language model; The video samples are input into the sign language visual recognition network model to obtain visual features, and then the corresponding visual intermediate features are obtained through multi-layer perceptron projection; Construct an auxiliary language sequence corresponding to the video sample, input it into the auxiliary language model, obtain language features, and then project it through a multi-layer perceptron to obtain the corresponding language intermediate features; Input the visual intermediate features and the language intermediate features into the cross-modal continuous temporal constraint module to calculate the inter-modal output difference index and the cross-modal continuous constraint loss; The training of the sign language visual recognition network model is completed by minimizing the cross-modal continuous constraint loss; The video to be recognized is input into the trained sign language visual recognition network model to obtain the recognition result.

2. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 1, characterized in that: The sign language visual recognition network model includes a feature extraction neural network layer for extracting visual features and a classification module for classifying the visual features.

3. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 2, characterized in that: The visual features are visual features extracted by any layer in the feature extraction neural network layer.

4. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 1, characterized in that: The auxiliary language model includes an encoder and a decoder.

5. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 4, characterized in that: The language features are language features extracted by any layer of the encoder.

6. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 1, characterized in that: The output difference index between the modalities is calculated using the following normalized dot product similarity function: in, represents the natural exponential function, represents a learnable scaling factor, Indicates The visual intermediate features corresponding to the frame video, Indicates Auxiliary intermediate variables corresponding to auxiliary sequence states, Indicates Auxiliary intermediate variables corresponding to auxiliary sequence states, represents the calculated similarity, and K represents the number of states of the auxiliary language sequence.

7. The sign language recognition method based on cross-modal continuous temporal constraints according to claim 1, characterized in that: The cross-modal continuous constraint loss is calculated using the CTC loss function: in, represents the cross-modal continuous constraint loss, represents the CTC loss function, S represents the output difference indicator matrix between modalities, The function generates a value from 0 to sequence of integers.

Images