A continuous sign language recognition method integrating cross-modal alignment auxiliary tasks

Abstract

The present invention discloses a method for continuous sign language recognition based on a cross-modal alignment auxiliary task, comprising: providing two cross-modal alignment auxiliary tasks, namely, a masked sign language word prediction task and a dynamic time warping contrastive learning task. The masked sign language word prediction task aligns sign language videos and sign language word sequences in a high-level semantic space, and the dynamic time warping contrastive learning task aligns sign language videos and sign language word sequences in a more dense temporal manner, providing more supervision in an end-to-end environment, alleviating the overfitting problem at the bottom layer of the network and enhancing model scalability. Simultaneously, both cross-modal alignment auxiliary tasks model the contextual information in the sign language word sequence. By aligning the sign language word sequence and the sign language video, the short-term spatiotemporal features in the context are optimized, addressing the problem that current end-to-end continuous sign language recognition methods generally ignore the mining of information in the sign language word sequence, thereby improving the recognition performance of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition, and in particular to a continuous sign language recognition method based on a cross-modal alignment auxiliary task. Background Art

[0002] With the widespread application of monocular cameras and the promotion of humanitarian care, the lives of the deaf and mute have attracted much attention. Communication between the deaf and mute and the normal people has gradually become the focus of attention. The problem of continuous sign language recognition based on machine learning has attracted a lot of attention. The deaf and mute communicate through sign language. Understanding or speaking sign language will help normal people better understand the needs of the deaf and mute and provide them with the necessary help.

[0003] Existing continuous sign language recognition networks can generally be divided into three modules: a visual module, a contextual module, and an alignment module. The visual module extracts frame-level or segment-level feature sequences from the sign language video input. The contextual module models the long-term relationships between the frame-level or segment-level feature sequences output by the visual module and outputs a feature sequence containing contextual information. The alignment module aligns the feature sequence and word sequence of the global context and provides supervision for model parameter optimization. Most recent work on continuous sign language recognition uses connectionist temporal classification (CTC) to align sign language videos and sign language word sequences. However, CTC suffers from a peaking phenomenon, where most of the global feature sequence is predicted as a null label, while only a few keyframes are attended to. When the dataset size is limited, this can lead to overfitting of the high-level contextual module and alignment module and insufficient training of the underlying parameters of the visual module, thus affecting the model's generalization ability and limiting the scalability and upper limit of model performance.

[0004] Recent continuous sign language recognition methods usually provide more supervision to the visual module through multi-stage optimization methods or auxiliary task learning, thereby alleviating the overfitting problem at the bottom layer of the network. Some continuous sign language recognition works have adopted multi-stage optimization methods for training. The multi-stage optimization method first uses connection temporal classification as the loss function for end-to-end training, and then uses the model for inference to obtain alignment proposals as frame-grained supervision information to optimize the visual module. The multi-stage optimization method can tap into the representational power of deep neural networks with limited data, but it requires multiple trainings and the design of objective functions at different stages. Some continuous sign language recognition works have adopted auxiliary task learning / multi-task learning methods to provide more supervision to the visual module under end-to-end training. Summary of the Invention

[0005] In view of the above, an object of the present invention is to provide a continuous sign language recognition method based on a cross-modal alignment auxiliary task to improve the accuracy of continuous sign language recognition.

[0006] To achieve the above-mentioned object of the invention, an embodiment provides a method for continuous sign language recognition based on a cross-modal alignment auxiliary task, comprising the following steps:

[0007] Prepare sign language videos and corresponding target sign language word sequences;

[0008] Constructing a training system, including an image feature extraction module, a short-term spatiotemporal feature extraction module, a long-term spatiotemporal feature extraction module, a masked sign language prediction module, a dynamic time warping contrastive learning module, and a connection temporal classification module. The video frame sequence corresponding to the sign language video is subjected to the image feature extraction module to obtain an image feature sequence. The image feature sequence is extracted by the short-term spatiotemporal feature extraction module to obtain a short-term spatiotemporal feature sequence. The short-term spatiotemporal feature sequence is introduced into the masked sign language prediction module to construct a visual feature sequence and participate in the mask prediction task of sign language words. The short-term spatiotemporal feature sequence is also introduced into the dynamic time warping contrastive learning module to construct a visual feature sequence and participate in the contrastive learning task of positive and negative sign language words. The short-term spatiotemporal feature sequence is subjected to the long-term spatiotemporal feature extraction module to extract a feature sequence. The short-term spatiotemporal feature sequence and the long-term spatiotemporal feature sequence are respectively subjected to the connection temporal classification module to calculate the activation classification probability and the connection temporal classification probability;

[0009] Construct loss functions, including prediction loss for mask prediction tasks, contrastive loss for contrastive learning tasks, distillation loss based on activation classification probabilities corresponding to short-term and long-term spatiotemporal feature sequences, and connection temporal classification loss based on connection temporal classification probabilities corresponding to short-term and long-term spatiotemporal feature sequences;

[0010] The loss function is used to optimize the parameters of the training system. After the parameter optimization is completed, the image feature extraction module, the short-term spatiotemporal feature extraction module, the long-term spatiotemporal feature extraction module, and the connection time series classification decoder are introduced to form a continuous sign language recognition model;

[0011] Continuous sign language recognition is performed using a continuous sign language recognition model.

[0012] In one embodiment, the image feature extraction module uses a residual convolutional neural network;

[0013] The short-term spatiotemporal feature extraction module adopts a one-dimensional convolutional neural network;

[0014] The long-term spatiotemporal feature extraction module adopts a bidirectional long short-term memory network.

[0015] In one embodiment, the masked sign language prediction module includes a first embedding layer, a first multi-head attention mechanism network, a second multi-head attention mechanism network, a third multi-head attention mechanism network and a linear layer. The mask sequence of the target sign language word sequence is masked and embedded by the first embedding layer to obtain a word embedding vector. The word embedding vector is introduced into the position embedding and input into the first multi-head attention mechanism network to obtain a sign language feature sequence of the fused context through calculation. The short-term spatiotemporal feature sequence is introduced into the position embedding and input into the second multi-head attention mechanism network to obtain a visual feature sequence of the fused context through calculation. The sign language feature sequence and the visual feature sequence are spliced ​​in the time dimension and input into the third multi-head attention mechanism network to obtain a multimodal feature sequence through fusion calculation. The first T3 features are extracted from the multimodal feature sequence and input into the linear layer for mapping calculation to obtain the prediction probability of the masked word.

[0016] In one embodiment, the prediction loss L for the mask prediction task is mgp Expressed as:

[0017]

[0018] Among them, Ⅱ represents the indicator function, i represents the word index, Represents the i-th word G i The return value is 1 when blocked, and 0 when not blocked. i ) represents the i-th word G i Mapping ordinal numbers in dictionaries, Represents the i-th word G i Mapping to D(G i ) probability value.

[0019] In one embodiment, the dynamic time warping contrastive learning module includes a negative sample generator, a second embedding layer, a third embedding layer, a fourth multi-head attention mechanism network, a fifth multi-head attention mechanism network and a sixth multi-head attention mechanism network. A single target sign language word sequence is used as a positive sample. The negative sample generator generates a negative sample based on the positive sample. The positive sample is embedded in the third embedding layer to obtain a word embedding vector. After position embedding is introduced into the word embedding vector, it is input into the fourth multi-head attention mechanism network and the sign language feature sequence of the positive sample is obtained through calculation. The negative sample is embedded in the fifth embedding layer to obtain a word embedding vector. After position embedding is introduced into the word embedding vector, it is input into the fifth multi-head attention mechanism network and the sign language feature sequence of the negative sample is obtained through calculation. After position embedding is introduced into the short-term spatiotemporal feature sequence, it is input into the sixth multi-head attention mechanism network and the visual feature sequence of the positive sample is obtained through calculation.

[0020] In one embodiment, the contrastive loss L of the contrastive learning task is dtw Expressed as:

[0021]

[0022] Among them, Q represents the visual feature sequence of the positive sample, C + represents the sign language features of the positive sample, C j represents the sign language features of the samples in set C, which consists of positive samples and their corresponding K negative samples. d(·) represents the distance function, and λ dtw Indicates the temperature coefficient that needs to be learned.

[0023] In one embodiment, the connection time series classification module includes an activation prediction unit, a connection time series classification alignment unit, and the short-term spatiotemporal feature sequence Y and the long-term spatiotemporal feature sequence Z are respectively activated in the activation prediction unit to obtain their corresponding prediction probabilities. and The connection time classification alignment unit defines a many-to-one function B, which can remove blanks or repeated words. For a given target sign language word sequence G, the feasible permutation path π∈Β -1 The connection temporal classification probabilities P(π|Z) and P(π|Y) of (G) are:

[0024]

[0025]

[0026] in, It represents the probability of the category at time t in the permutation path π predicted by Z at the tth time step, It represents the probability of the category at time t in the permutation path π predicted by Y at the tth time step, where T is the total time step.

[0027] In one embodiment, the distillation loss L distill Expressed as:

[0028]

[0029] Here, KL(·) represents the KL divergence.

[0030] In one embodiment, the connection temporal classification loss L ctc Expressed as:

[0031]

[0032] In one embodiment, the connection time series classification decoder includes a parameter-optimized activation prediction unit, which uses the parameter-optimized activation prediction unit to predict the activation classification probability of the long-term spatiotemporal feature sequence, and introduces a beam search method to predict the activation classification probability of the long-term spatiotemporal feature sequence. Decode to get the predicted sign language word sequence, including: beam search method retains the first Ta time steps The beam size prefix substrings with the largest probability are calculated at the Ta+1 time step. The beam search method uses the activation classification prediction probability of the Ta+1 time step to calculate the new probability of adding different categories to the end of each prefix substring. The Beta function in the connection time series classification is used to remove duplicates and empty classes for each prefix substring, and the same new prefix substrings and the corresponding new probabilities are merged into one prefix substring. Similarly, the beam search method can find a suitable solution at the T2 time step to obtain the predicted sign language word sequence.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] Two cross-modal alignment auxiliary tasks are provided, namely the masked sign language word prediction task and the dynamic time warping contrastive learning task. The masked sign language word prediction task aligns sign language videos and sign language word sequences in a high-level semantic space, and the dynamic time warping contrastive learning task aligns sign language videos and sign language word sequences in a more dense manner in time sequence, providing more supervision in an end-to-end manner, alleviating the overfitting problem at the bottom layer of the network and enhancing the scalability of the model.

[0035] Furthermore, both cross-modal alignment auxiliary tasks model the contextual information in sign language word sequences. By aligning sign language word sequences and sign language videos to optimize the short-term spatiotemporal features in the context, they solve the problem that current end-to-end continuous sign language recognition methods usually ignore the mining of information in sign language word sequences, thereby improving the recognition performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 is a flowchart of a continuous sign language recognition method based on a cross-modal alignment auxiliary task provided by an embodiment;

[0038] Figure 2 is a schematic structural diagram of a training system provided in an embodiment;

[0039] Figure 3 is a schematic structural diagram of a masked sign language prediction module provided in an embodiment;

[0040] Figure 4 2 is a schematic diagram of the structure of the dynamic time warping comparative learning module provided in the embodiment;

[0041] Figure 5 is a schematic structural diagram of a continuous sign language recognition model provided by an embodiment;

[0042] Figure 6 is a visualization diagram of the prediction scores of a typical example provided in the embodiment;

[0043] Figure 7 3 is a comparison chart of the reasoning recognition results of typical examples provided in the embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0045] Figure 1 Flowchart of the continuous sign language recognition method based on the cross-modal alignment auxiliary task provided by the embodiment. Figure 1 As shown, the embodiment provides a continuous sign language recognition method based on a cross-modal alignment auxiliary task, comprising the following steps:

[0046] Step 1: Prepare sign language video and corresponding target sign language word sequence.

[0047] In the embodiment, taking the sign language video 04July_2010_Sunday_tagesschau_default-7 as an example, the sign language video is saved in the format of video frame images, and all images are resized to 256×256 pixels. During the training phase, the input sign language video is randomly cropped in the spatial dimension, and the cropped video resolution is 224×224. The video is horizontally flipped with a probability of 50%, and the length of the video is adjusted to 80% to 120% of the original length. When the target length is shorter than the original length, the sampling method without replacement is used. When the target length is longer than the original length, the sampling method with replacement is used. The video frame images are normalized using ImageNet color, that is, the mean is (0.485, 0.456, 0.406) and the variance is (0.229, 0.224, 0.225).

[0048] In the embodiment, a target sign language word sequence corresponding to the sign language video is also prepared, and the sign language video and the corresponding target sign language word sequence are used as a sample data for system training.

[0049] Step 2: Build the training system and loss function.

[0050] In the embodiment, in order to make the underlying parameters trained sufficiently to improve the generalization ability of the model, and at the same time enhance the scalability of the model and the upper limit of the model performance, two cross-modal alignment auxiliary tasks are proposed, namely the masked sign language word prediction task and the dynamic time warping contrastive learning task. By modeling the contextual information of the sign language word sequence, the sign language video and the sign language word sequence are aligned in the high-level semantic space and time sequence, respectively, providing more supervision for the visual information in the case of end-to-end training. Based on this, Figure 2 As shown in the figure, the constructed training system includes an image feature extraction module, a short-term spatiotemporal feature extraction module, a long-term spatiotemporal feature extraction module, a masked sign language prediction module, a dynamic time warping contrast learning module, and a connection time series classification module.

[0051] The image feature extraction module uses a residual convolutional neural network to extract the image features of the input video frame sequence. Specifically, the ResNet18 network can be used. Each frame image in the video S is input into the ResNet18 network, and the output of the average pooling layer of the last layer of the ResNet18 network is taken as the spatial feature of each frame image to obtain a one-dimensional image feature sequence at the frame level. The characteristic length C1 is 512.

[0052] The short-term spatiotemporal feature extraction module uses a convolutional neural network to extract the short-term spatiotemporal feature sequence Y based on the one-dimensional image feature sequence X. Specifically, a one-dimensional convolutional neural network can be used, which is composed of two one-dimensional convolutional layers Conv1D with ReLU activation function and batch normalization layer BN, and a maximum pooling layer MaxPooling with a stride of 2. Its structural formula is:

[0053] X 1 =ReLu(BN(Conv1D(X)))

[0054] X 2 =MaxPooling(X 1 )

[0055] Y = ReLu(BN(Conv1D(X 2 ))

[0056] In a one-dimensional convolutional neural network, the maximum pooling layer is used to reduce the length of the image feature sequence, and the relationship between adjacent frame-level image features is modeled through convolution in the time dimension to obtain a short-term spatiotemporal feature sequence. The hidden feature dimension C2 is 2048, the convolution kernel size of the temporal convolution layer is 2, the padding is 1, the number of convolution kernels is 2048, and the stride of the maximum pooling layer is 2.

[0057] The long-term spatiotemporal feature extraction module is used to extract the long-term spatiotemporal feature sequence based on the short-term spatiotemporal feature sequence Y. Specifically, the long-term spatiotemporal feature extraction module can use a network stacked with two layers of bidirectional long short-term memory (BiLSTM), where the dropout is set to 0.1, and the long-term spatiotemporal feature sequence is obtained by extracting the input Y through the two-layer BiLSTM.

[0058] The masked sign language prediction module is used to construct a visual feature sequence using the introduced short-term spatiotemporal feature sequence to participate in the mask prediction task of sign language words, so as to align the sign language word sequence and sign language video in a higher-level semantic space. Figure 3 As shown, the masked sign language prediction module consists of the first embedding layer, the first multi-head attention network, the second multi-head attention network, the third multi-head attention network, and the linear layer. The target sign language word sequence G is masked, and each sign language word in G is represented by r m The mask probability is replaced by the [MASK] identifier, and the [CLS] identifier is added at the beginning and the [SEP] identifier is added at the end to form a mask sequence where r m The value of is set to 0.9. The mask sequence is embedded in the first embedding layer to obtain a word embedding vector of length C2=2048, which is then introduced into the position embedding to obtain the embedding vector of the sign language word sequence. The embedding vector O is input into the first multi-head attention mechanism network and converted into a latent space shared with the sign language video to obtain the sign language feature sequence H fused with the context. g , the short-term spatiotemporal feature sequence Y is introduced into the position embedding and input into the second multi-head attention mechanism network and the visual feature sequence H of the fusion context is obtained after calculation v , the sign language feature sequence and the visual feature sequence are spliced ​​in the time dimension to obtain H = [H g ;H v ],in, The H is input into the third multi-head attention mechanism network, and the multimodal feature sequence is obtained by fusing the features between different modalities. Extract the first T3 sign language word feature sequences that incorporate sign language video features from the multimodal feature sequence P Input the linear layer for mapping calculation to obtain the predicted probabilities of sign language words, [CLS] identifiers, and [SEP] identifiers.

[0059] In this embodiment, the prediction loss L of the mask prediction task is calculated based on the prediction probability of the mask word. mgp , expressed as:

[0060]

[0061] Among them, Ⅱ represents the indicator function, i represents the word index, Represents the i-th word G i The return value is 1 when blocked, and 0 when not blocked. i ) represents the i-th word G i Mapping ordinal numbers in dictionaries, Represents the i-th word G i Mapping to D(G i ) probability value.

[0062] The dynamic time warping contrastive learning module is used to construct a visual feature sequence using the introduced short-term spatiotemporal feature sequence to participate in contrastive learning, so as to align the sign language video and the target sign language word sequence in a more dense manner in terms of time sequence. The dynamic time warping contrastive learning task minimizes the differentiable dynamic time warping distance between the short-term spatiotemporal feature sequence Y of the sign language video and the target sign language word sequence, thereby aligning the target sign language word sequence G and the sign language video S. In order to prevent the model from collapsing, negative samples are introduced. Figure 4 As shown, the dynamic time warping contrastive learning module includes a negative sample generator, a second embedding layer, a third embedding layer, a fourth multi-head attention mechanism network, a fifth multi-head attention mechanism network, and a sixth multi-head attention mechanism network. A single target sign language word sequence is used as a positive sample. The negative sample generator generates K negative samples based on the positive sample to form a negative sample set A. The positive sample and the negative sample set are added to obtain a set of candidate sign language word sequences.

[0063] In order to align the sequences of the two modalities in a fine-grained manner, the contextual information of the candidate sign language word sequences is modeled to collect Taking the positive sample G in the example, the positive sample G is embedded in the third embedding layer to obtain a word embedding vector. The word embedding vector is introduced into the position embedding and then input into the fourth multi-head attention mechanism network for context information modeling. At the same time, it is converted into a shared latent space to obtain the sign language feature sequence C of the positive sample. + , All negative samples in the above operation are also processed to obtain the sign language feature sequence of the negative sample. The sign language feature sequence of the positive sample and the negative sample is composed of C = *C1,...,C j ,...,C K ,C K+1},in, The short-term spatiotemporal feature sequence Y is introduced into the position embedding and input into the sixth multi-head attention mechanism network to perform context information modeling and convert it to the same as C j The shared latent space obtains the visual feature sequence of the positive sample

[0064] Based on the above contrastive learning, cross entropy is used as the loss of dynamic time warping contrastive learning to bring Q and C closer. +distance, distance Q from other negative samples, contrast loss L dtw for:

[0065]

[0066] Among them, d(·) represents the distance function, λ dtw Indicates the temperature coefficient that needs to be learned.

[0067] In an embodiment, all of the above-mentioned multi-head attention mechanism networks are composed of a stack of multiple multi-head attention mechanism sub-layers and feedforward neural network sub-layers. Specifically, the multi-head attention mechanism networks are composed of 2 layers of multi-head attention mechanism sub-layers, each layer has 8 self-attention heads, and the hidden layer feature dimension is 2048.

[0068] The connection temporal classification module calculates the connection temporal classification loss and distillation loss by aligning Y, Z and the target sign language word sequence G. The connection temporal classification module includes an activation prediction unit and a connection temporal classification alignment unit. The short-term spatiotemporal feature sequence Y and the long-term spatiotemporal feature sequence Z are activated in the activation prediction unit to obtain their corresponding prediction probabilities. and The activation calculation is expressed as:

[0069]

[0070] Based on the predicted probability and Construct the distillation loss L distill Expressed as:

[0071]

[0072] Here, KL(·) represents the KL divergence, which provides more supervision for the underlying network by distilling long-term spatiotemporal features to short-term spatiotemporal features.

[0073] The connection time classification alignment unit defines a many-to-one function B, which can remove blanks or repeated words. For a given target sign language word sequence G, the feasible permutation path π∈Β -1 The connection temporal classification probabilities P(π|Z) and P(π|Y) of (G) are:

[0074]

[0075]

[0076] in, It represents the probability of the category at time t in the permutation path π predicted by Z at the tth time step, =(\begin{element of t}) represents the probability of the class at time t in the permutation path π predicted based on Y at time step t. The class at time 0 represents the empty class, and the classes at other time steps represent sign language words. L2 regularization is applied to the vector of each class to reduce the impact of class imbalance on performance. T is the total number of time steps.

[0077] The connection temporal classification loss L calculated based on the above connection temporal classification probability ctc Expressed as:

[0078]

[0079] Step 3: Use the loss function to optimize the training system and extract and build a continuous sign language recognition model.

[0080] In this embodiment, the training system parameters are optimized using sample data and a loss function. During parameter optimization, a dropout value of 0.1 is used. Adam is used as the optimizer, with a weight decay of 1e-4. The initial learning rate is set to 1e-4. Two A100 graphics cards with 40GB of video memory are used, with a batch size of 2 for each card and a training epoch number of 80.

[0081] After the parameter optimization is completed, Figure 5 As shown in the figure, a parameter-optimized image feature extraction module, a short-term spatiotemporal feature extraction module, and a long-term spatiotemporal feature extraction module are used, and a connection temporal classification decoder is introduced to form a continuous sign language recognition model.

[0082] In the embodiment, the connection time series classification decoder includes a parameter-optimized activation prediction unit, which uses the parameter-optimized activation prediction unit to predict the activation classification probability of the long-term spatiotemporal feature sequence, and also introduces a beam search method to predict the activation classification probability of the long-term spatiotemporal feature sequence. Decode to get the predicted sign language word sequence, including: beamsearch method retains the first Ta time steps The beam size prefix substrings with the largest probability are calculated at the Ta+1 time step. The beam search method uses the activation classification prediction probability of the Ta+1 time step to calculate the new probability of adding different categories to the end of each prefix substring. The Beta function in the connection time series classification is used to remove duplicates and empty classes for each prefix substring, and the same new prefix substrings and the corresponding new probabilities are merged into one prefix substring. Similarly, the beam search method can find a suitable solution at the T2 time step to obtain the predicted sign language word sequence.

[0083] Step 4: Perform continuous sign language recognition using a continuous sign language recognition model.

[0084] In the embodiment, after obtaining the continuous sign language recognition model constructed in step 3, the continuous sign language recognition model can be used to perform continuous sign language recognition. The specific process includes:

[0085] After the sign language video is framed, it is input into the continuous sign language recognition model. It passes through the image feature extraction module, the short-term spatiotemporal feature extraction module, and the long-term spatiotemporal feature extraction module in sequence. The long-term spatiotemporal feature sequence is extracted and input into the connected temporal classification decoder. After the activation classification probability of the long-term spatiotemporal feature sequence is predicted, the activation classification probability of the long-term spatiotemporal feature sequence is predicted using the beam search method. Decoding is performed to obtain the predicted sign language word sequence.

[0086] Figure 6 and Figure 7 The embodiment shown qualitatively verifies the continuous sign language recognition method proposed by the present invention. Figure 6 The figure shows the prediction scores for each time step of the 04July_2010_Sunday_tagesschau_default-7 sign language video. Curves of different colors represent the prediction probabilities of sign language words of the same color. The proposed method predicts "NICHT-HABEN" and "WEHEN" with higher probabilities at multiple time steps, while the baseline method predicts these two sign language words in only one time step, demonstrating the superior robustness of the proposed method.

[0087] Figure 7 The figure shows the 04July_2010_Sunday_tagesschau_default-7 sign language video and sign language word annotations, as well as a comparison of the inference results of the proposed method and a baseline method. Underlined areas indicate mismatches with the sign language word annotations. This intuitive comparison shows that the proposed method can more accurately perform continuous sign language recognition than the baseline method, qualitatively demonstrating the effectiveness of the proposed method. This demonstrates that the proposed method can optimize the intermediate features of the continuous sign language recognition network by aligning the sign language video sequence and the sign language word sequence in semantic space and time, thereby improving the network's continuous sign language recognition performance.

[0088] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A continuous sign language recognition method based on a cross-modal alignment auxiliary task, characterized in that: The following steps are involved: Prepare sign language videos and corresponding target sign language word sequences; Constructing a training system, including an image feature extraction module, a short-term spatiotemporal feature extraction module, a long-term spatiotemporal feature extraction module, a masked sign language prediction module, a dynamic time warping contrastive learning module, and a connection temporal classification module. The video frame sequence corresponding to the sign language video is subjected to the image feature extraction module to obtain an image feature sequence. The image feature sequence is extracted by the short-term spatiotemporal feature extraction module to obtain a short-term spatiotemporal feature sequence. The short-term spatiotemporal feature sequence is introduced into the masked sign language prediction module to construct a visual feature sequence and participate in the mask prediction task of sign language words. The short-term spatiotemporal feature sequence is also introduced into the dynamic time warping contrastive learning module to construct a visual feature sequence and participate in the contrastive learning task of positive and negative sign language words. The short-term spatiotemporal feature sequence is subjected to the long-term spatiotemporal feature extraction module to extract a feature sequence. The short-term spatiotemporal feature sequence and the long-term spatiotemporal feature sequence are respectively subjected to the connection temporal classification module to calculate the activation classification probability and the connection temporal classification probability; Among them, the masked sign language prediction module includes a first embedding layer, a first multi-head attention mechanism network, a second multi-head attention mechanism network, a third multi-head attention mechanism network and a linear layer. The mask sequence of the target sign language word sequence is masked and embedded in the first embedding layer to obtain a word embedding vector. The word embedding vector is introduced into the position embedding and input into the first multi-head attention mechanism network to obtain a sign language feature sequence with fusion context after calculation. The short-term spatiotemporal feature sequence is introduced into the position embedding and input into the second multi-head attention mechanism network to obtain a visual feature sequence with fusion context after calculation. The sign language feature sequence and the visual feature sequence are spliced ​​in the time dimension and input into the third multi-head attention mechanism network to obtain a multimodal feature sequence through fusion calculation. The first T3 features are extracted from the multimodal feature sequence and input into the linear layer for mapping calculation to obtain the prediction probability of the masked word. The dynamic time warping contrastive learning module includes a negative sample generator, a second embedding layer, a third embedding layer, a fourth multi-head attention mechanism network, a fifth multi-head attention mechanism network, and a sixth multi-head attention mechanism network. A single target sign language word sequence is used as a positive sample. The negative sample generator generates a negative sample based on the positive sample. The positive sample is embedded in the third embedding layer to obtain a word embedding vector. After position embedding is introduced into the word embedding vector, it is input into the fourth multi-head attention mechanism network and the sign language feature sequence of the positive sample is obtained through calculation. The negative sample is embedded in the fifth embedding layer to obtain a word embedding vector. After position embedding is introduced into the word embedding vector, it is input into the fifth multi-head attention mechanism network and the sign language feature sequence of the negative sample is obtained through calculation. The short-term spatiotemporal feature sequence is introduced into the position embedding and input into the sixth multi-head attention mechanism network and the visual feature sequence of the positive sample is obtained through calculation. Construct loss functions, including prediction loss for mask prediction tasks, contrastive loss for contrastive learning tasks, distillation loss based on activation classification probabilities corresponding to short-term and long-term spatiotemporal feature sequences, and connection temporal classification loss based on connection temporal classification probabilities corresponding to short-term and long-term spatiotemporal feature sequences; The loss function is used to optimize the parameters of the training system. After the parameter optimization is completed, the image feature extraction module, the short-term spatiotemporal feature extraction module, the long-term spatiotemporal feature extraction module, and the connection time series classification decoder are introduced to form a continuous sign language recognition model; Continuous sign language recognition is performed using a continuous sign language recognition model.

2. The continuous sign language recognition method based on the cross-modal alignment auxiliary task according to claim 1 is characterized in that: The image feature extraction module adopts a residual convolutional neural network; The short-term spatiotemporal feature extraction module adopts a one-dimensional convolutional neural network; The long-term spatiotemporal feature extraction module adopts a bidirectional long short-term memory network.

3. The continuous sign language recognition method based on the cross-modal alignment auxiliary task according to claim 1 is characterized in that: The prediction loss L of the mask prediction task mgp Expressed as: Among them, Ⅱ represents the indicator function, i represents the word index, Represents the i-th word G i The return value is 1 when blocked, and 0 when not blocked. i ) represents the i-th word G i Mapping ordinal numbers in dictionaries, Represents the i-th word G i Mapping to D(G i ) probability value.

4. The continuous sign language recognition method based on the cross-modal alignment auxiliary task according to claim 1 is characterized in that: The contrastive loss L for the contrastive learning task dtw Expressed as: Among them, Q represents the visual feature sequence of the positive sample, C + represents the sign language features of the positive sample, C j represents the sign language features of the samples in set C, which consists of positive samples and their corresponding K negative samples. d(·) represents the distance function, and λ dtw Indicates the temperature coefficient that needs to be learned.

5. The continuous sign language recognition method based on cross-modal alignment auxiliary task according to claim 1 is characterized in that: The connection time series classification module includes an activation prediction unit and a connection time series classification alignment unit. The short-term spatiotemporal feature sequence Y and the long-term spatiotemporal feature sequence Z are respectively activated and predicted in the activation prediction unit to obtain their corresponding prediction probabilities. and The connection time classification alignment unit defines a many-to-one function B that removes blanks or repeated words. For a given target sign language word sequence G, the feasible permutation path π∈Β -1 The connection temporal classification probabilities P(π|Z) and P(π|Y) of (G) are: in, It represents the probability of the category at time t in the permutation path π predicted by Z at the tth time step, It represents the probability of the category at time t in the permutation path π predicted by Y at the tth time step, where T is the total time step.

6. The continuous sign language recognition method based on cross-modal alignment auxiliary task according to claim 5, characterized in that: The distillation loss L distill Expressed as: Here, KL(·) represents the KL divergence.

7. The continuous sign language recognition method based on cross-modal alignment auxiliary task according to claim 5, characterized in that: The connection temporal classification loss L ctc Expressed as:

8. The continuous sign language recognition method based on cross-modal alignment auxiliary task according to claim 1, characterized in that: The connection time series classification decoder includes a parameter-optimized activation prediction unit, which uses the parameter-optimized activation prediction unit to predict the activation classification probability of the long-term spatiotemporal feature sequence, and introduces the beam search method to predict the activation classification probability of the long-term spatiotemporal feature sequence. Decode to get the predicted sign language word sequence, including: beam search method retains the first Ta time steps The beam size prefix substrings with the largest probability are calculated at the Ta+1 time step. The beam search method uses the activation classification prediction probability of the Ta+1 time step to calculate the new probability of adding different categories to the end of each prefix substring. The Beta function in the connection time series classification is used to remove duplicates and empty classes for each prefix substring, and the same new prefix substrings and the corresponding new probabilities are merged into one prefix substring. Similarly, the beam search method can find a suitable solution at the T2 time step to obtain the predicted sign language word sequence.

Images