Continuous sign language recognition method based on time-space multi-attention mechanism fusion

Abstract

The invention relates to the technical field of image information processing, in particular to a continuous sign language recognition method based on time-space multi-attention mechanism fusion, which comprises the following steps: acquiring a sign language video to be recognized; the sign language video to be recognized is input into a preset sign language recognition model, a continuous sign language recognition result is output, the sign language recognition model is obtained through training of a training set, the training set comprises a plurality of sign language videos, and the sign language recognition model is constructed based on a neural network in combination with a space-time multi-attention mechanism. The sign language recognition model constructed by fusing the space-time attention mechanism can better integrate and enhance the features of the space-time and each channel, and focus and highlight the key space-time features through the convolution attention mechanism under the condition of almost not increasing the computing power overhead, thereby better realizing event positioning and continuous sign language recognition of the sign language video.

Description

Technical Field

[0001] The present invention relates to the technical field of image information processing, and in particular to a continuous sign language recognition method based on the fusion of spatiotemporal multi-attention mechanisms. Background Art

[0002] Sign language video is a medium presentation form of continuous sign language, which enables disabled people and healthy people to communicate without obstacles. In continuous sign language recognition, how to perform effective feature extraction is very critical. In order to extract more effective spatial, channel and temporal features, previous methods usually use additional input auxiliary feature clues for supervision. Although it can enhance the extraction effectiveness of the feature extractor to a certain extent and reduce the word error rate in continuous sign language recognition, it will also pay a huge computing power overhead. In addition to increasing the GPU memory consumption, it will also lead to longer model training time. A single spatial attention mechanism is used to enhance spatial features, and the channel attention and temporal attention mechanisms are not simultaneously integrated for more comprehensive feature association, and the spatiotemporal feature information of the input sign language video is not fully mined. Therefore, a continuous sign language recognition method based on the fusion of spatiotemporal multi-attention mechanisms is urgently needed. Summary of the invention

[0003] The purpose of the present invention is to provide a continuous sign language recognition method based on the fusion of spatiotemporal multi-attention mechanisms. By integrating the sign language recognition model constructed by the spatiotemporal attention mechanism, the spatiotemporal and channel features are integrated and enhanced, and the key spatiotemporal features are focused and highlighted through the convolutional attention mechanism without almost increasing the computing power overhead, so as to better realize event positioning and continuous sign language recognition in sign language videos.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A continuous sign language recognition method based on spatiotemporal multi-attention mechanism fusion, comprising:

[0006] Get the sign language video to be recognized;

[0007] The sign language video to be recognized is input into a preset sign language recognition model, and a continuous sign language recognition result is output, wherein the sign language recognition model is obtained by training with a training set, the training set includes several sign language videos, and the sign language recognition model is constructed based on a neural network combined with a spatiotemporal multi-attention mechanism.

[0008] Optionally, before training the sign language recognition model using the training set, the method further includes preprocessing the sign language video in the training set, wherein the preprocessing includes:

[0009] scaling the sign language video, randomly cropping the scaled sign language video, and randomly horizontally flipping the cropped sign language video to obtain a preliminarily processed sign language video;

[0010] The sign language video after preliminary processing is randomly divided into video groups, and re-joined into preset sizes to obtain a final sign language video, thus completing the pre-processing.

[0011] Optionally, the sign language recognition model includes: a feature extraction module, a timing module, and a classification prediction module, wherein the feature extraction module is used to extract visual features of the sign language video to be recognized, and send them to the timing module and the classification prediction module respectively; the timing module is used to extract context features based on the visual features, and send them to the classification prediction module; the classification prediction module is used to perform sign language category probability prediction based on the visual features and the context features respectively, and output continuous sign language recognition results.

[0012] Optionally, the feature extraction module includes: a first convolution unit, a channel attention unit, a spatial attention unit, a temporal attention unit, and a second convolution unit. The sign language video to be recognized first passes through the first convolution unit, and then passes through a first branch composed of the channel attention unit and the spatial attention unit and a second branch composed of the temporal attention unit, and finally passes through the second convolution unit to output the visual features.

[0013] Optionally, the timing module uses a bidirectional long short-term memory network to extract context features based on the visual features; the classification prediction module uses a main classifier and an auxiliary classifier to share weights to perform sign language category probability prediction based on the visual features and the context features.

[0014] Optionally, the training process of training the sign language recognition model using the training set includes:

[0015] Step 1: extract a video segment from the preprocessed training set, input it into the sign language recognition model, and obtain a first prediction result and a second prediction result obtained by the classification prediction module for predicting the probability of the sign language category based on the visual features and the context features respectively;

[0016] Step 2: Calculate the loss based on the first prediction result and the second prediction result to obtain a first loss result and a second loss result;

[0017] Step 3: updating the parameters of the sign language recognition model based on the first loss result and the second loss result to obtain an updated sign language recognition model;

[0018] Step 4: re-extract the video segment, repeat steps 1 to 3, and update the parameters of the updated sign language recognition model. If the model converges, the training is completed.

[0019] Optionally, the training process of training the sign language recognition model through the training set also includes: if the model has not converged until the video segments in the preprocessed training set are extracted, the order of the video segments in the preprocessed training set is shuffled, and steps 1 to 3 are repeated until the model converges.

[0020] Optionally, the loss function for performing the loss calculation is:

[0021]

[0022] In the formula, For the final loss, is the main loss for CTC decoding, is the auxiliary loss for CTC decoding, is the knowledge distillation loss, μ is a hyperparameter, p is the recognition success probability of continuous sign language events, l is the actual recognition result of continuous sign language events, F is the feature sequence of visual features or context features, θ is the global parameter of the model, v is the auxiliary classifier, KL is the KL divergence calculation, Z is the feature sequence of context features after the classification prediction module, is the feature sequence of the visual features after the classification prediction module, and τ is the high temperature parameter.

[0023] The beneficial effects of the present invention are:

[0024] The present invention adopts a fusion-based multi-dimensional visual attention feature extractor to capture the visual representation of each frame, and uses 1D CNN and BiLSTM for short-term and long-term temporal modeling respectively. Through multi-module and multi-level processing, composite features (temporal features, spatial features and channel features) are extracted, timing prediction and word alignment are performed on the video frame sequence of continuous sign language, thus realizing a new sign language video event localization algorithm.

[0025] The present invention designs and adds three low-computing-power-overhead convolutional attention modules in the feature extractor, namely, the channel attention module (CRAM), the spatial attention module (SRAM) and the temporal attention module (TRAM), and structures the three as a staggered layout mode in which the channel and spatial attention are combined in series and then in parallel with the temporal attention, so as to enhance and fuse the spatiotemporal features, and provide good deep features for subsequent sign language recognition prediction; the sign language recognition model after integrating the spatiotemporal attention mechanism can better integrate and enhance the spatiotemporal and channel features, and focus on and highlight the key spatiotemporal features through the convolutional attention mechanism with almost no increase in computing power, so as to better realize event positioning and continuous sign language recognition of sign language videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 This is a flow chart of a method for continuous sign language recognition based on the fusion of spatiotemporal multi-attention mechanisms according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of a network structure of a sign language recognition model according to an embodiment of the present invention;

[0029] Figure 3 This is a flow chart of the sign language recognition model training according to an embodiment of the present invention;

[0030] Figure 4 This is a flow chart of updating the parameters of the sign language recognition model according to an embodiment of the present invention.

[0031] Figure 5 A schematic diagram of the network structure of a temporal attention unit according to an embodiment of the present invention;

[0032] Figure 6 Schematic diagram of the channel attention unit network structure of an embodiment of the present invention;

[0033] Figure 7 Schematic diagram of the spatial attention unit network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] This embodiment provides a continuous sign language recognition method based on the fusion of spatiotemporal multi-attention mechanisms. Figure 1-Figure 7 ,include:

[0037] Get the sign language video to be recognized;

[0038] The sign language video to be recognized is input into a preset sign language recognition model, and a continuous sign language recognition result is output, wherein the sign language recognition model is obtained by training a training set, the training set includes several sign language videos, and the sign language recognition model is constructed based on a neural network combined with a spatiotemporal multi-attention mechanism.

[0039] Specifically, a large number of sign language video data sets are obtained by shooting or directly intercepting existing videos, and a sign language recognition model is trained using the large number of sign language video data sets. The sign language recognition model is used to identify continuous sign language actions in the target video, wherein the sign language recognition model is constructed by using a neural network combined with a spatiotemporal multi-attention mechanism, and the spatiotemporal multi-attention mechanism refers to channel attention, spatial attention, and time attention. In this embodiment, the neural network uses 1D CNN and BiLSTM. By adopting the visual representation of each frame, 1D CNN and BiLSTM are used for short-term and long-term time modeling respectively. Then, through multi-module and multi-level processing, the video frame sequence of continuous sign language is subjected to composite feature (time feature, spatial feature and channel feature) extraction, time series prediction and word alignment. Next, the initial visual feature extraction of continuous sign language recognition is strengthened by adding a spatiotemporal attention module, thereby improving the accuracy of continuous sign language recognition.

[0040] Furthermore, before training the sign language recognition model with the training set, the sign language video in the training set is preprocessed, wherein the preprocessing includes:

[0041] scaling the sign language video, randomly cropping the scaled sign language video, and randomly horizontally flipping the cropped sign language video to obtain a preliminarily processed sign language video;

[0042] The sign language video after preliminary processing is randomly divided into video groups and re-joined at a preset size to obtain the final sign language video and complete the pre-processing.

[0043] Specifically, in this embodiment, each frame of the sign language video selected in the training set is scaled and randomly cropped in the middle area of ​​the image, so that each image in the database has a consistent size, with a length and width of h and w respectively, and a number of channels of c. After cropping and scaling, the image is randomly flipped horizontally to obtain an image;

[0044] In the preprocessed data, each object is randomly divided into a video group (batch) consisting of n videos. If the integer of all images cannot divide n, the remainder videos are grouped together. The n videos in each batch are concatenated on the batch dimension to obtain n new image segments of size (t, c, w, h), where t is the maximum number of frames in the n videos.

[0045] Furthermore, the sign language recognition model includes: a feature extraction module, a timing module, and a classification prediction module, wherein the feature extraction module is used to extract visual features of the sign language video to be recognized, and send them to the timing module and the classification prediction module respectively; the timing module is used to extract context features based on visual features, and send them to the classification prediction module; the classification prediction module is used to predict the sign language category probability based on visual features and context features respectively, and output continuous sign language recognition results.

[0046] Among them, the feature extraction module includes: a first convolution unit, a channel attention unit, a spatial attention unit, a temporal attention unit, and a second convolution unit. The sign language video to be recognized first passes through the first convolution unit, and then passes through the first branch composed of the channel attention unit and the spatial attention unit and the second branch composed of the temporal attention unit, and finally passes through the second convolution unit to output visual features.

[0047] The timing module uses a bidirectional long short-term memory network to extract contextual features based on visual features; the classification prediction module uses the main classifier and auxiliary classifier to share weights to predict the sign language category probability based on visual features and contextual features.

[0048] Specifically, this embodiment uses a deep learning framework to build an end-to-end sign language recognition model based on the structures and parameters of the feature extraction module, the timing module, and the classification prediction module.

[0049] The feature extraction module consists of a two-dimensional convolution layer, a maximum pooling layer, an average pooling layer, 8 layers of MAFN-Block and a fully connected layer. Each layer of MAFN-Block consists of two layers of two-dimensional convolution, channel attention module, spatial attention module and time attention module. First, the data passes through the first layer of two-dimensional convolution and passes through two branches in parallel. The first branch is composed of channel attention module and spatial attention module in sequence, and the second branch is composed of time attention module. After passing through the two branches, the data is combined by addition and then passed through the second layer of two-dimensional convolution to obtain the main output. The original data also needs to pass through a residual branch. Except for the 1st and 2nd layers, the residual branch of each layer is composed of a two-dimensional convolution layer. The data passing through the main branch and the residual branch are combined by addition to obtain the total output.

[0050] The parameters of the two-dimensional convolution layer of the feature extraction module are: the number of convolution kernels is 64, the size is 7x7, the step size is 2, and the edge filling is a constant 0 filling of 3 layers; the parameters of the maximum pooling layer are: the pooling kernel size is 3, the pooling step size is 2, the edge filling is a constant 0 filling of 1 layer, and the expansion factor is 1; the parameters of the average pooling layer are: the pooling kernel size is 7, and the pooling step size is 7; the structure of every two layers of the MAFN-Block layer is the same, and the first layer of the two-dimensional convolution parameters of the 1st and 2nd layers of MAFN-Block are: the number of convolution kernels is 64, the size is 3x3, the step size is 1, and the edge filling is a constant 0 filling of 1 layer; the temporal attention module layer consists of two one-dimensional convolutions:

[0051] The first layer of convolution parameters: the number of convolution kernels is 16, the size is 3, the step size is 1, and the edge padding is a constant 0 padding of 1 layer;

[0052] The second layer of convolution parameters: the number of convolution kernels is 64, the size is 1, and the step size is 1; the channel attention module consists of two fully connected layers with 4 and 64 hidden units respectively, and the spatial attention module consists of a two-dimensional convolution with 1 convolution kernel, 7x7 size, 1 step size, and 3 layers of constant 0 padding on the edge; the second layer of two-dimensional convolution parameters: the number of convolution kernels is 64, the size is 3x3, the step size is 1, and the edge padding is 1 layer of constant 0 padding; the first layer of two-dimensional convolution parameters of the 3rd and 4th layers of MAFN-Block: the number of convolution kernels is 128, the size is 3x3, the step size is 2, and the edge padding is 1 layer of constant 0 padding; the temporal attention module layer consists of two one-dimensional convolutions: the first layer of convolution parameters: the number of convolution kernels is 32, size 3, stride 1, edge padding is 1 layer of constant 0 padding; the second layer of convolution parameters: the number of convolution kernels is 128, size 1, stride 1; the channel attention module consists of two fully connected layers with 8 and 128 hidden units respectively, and the spatial attention module consists of a two-dimensional convolution with 1 convolution kernel, size 7x7, stride 1, and edge padding is 3 layers of constant 0 padding; the second layer of two-dimensional convolution parameters: the number of convolution kernels is 128, size 3x3, stride 1, and edge padding is 1 layer of constant 0 padding; the first layer of two-dimensional convolution parameters of the 5th and 6th layers of MAFN-Block: the number of convolution kernels is 256, size 3x3, stride 2, and edge padding is 1 layer of constant 0 padding; time The spatial attention module layer consists of two one-dimensional convolutions: the first layer convolution parameters: the number of convolution kernels is 64, the size is 3, the step size is 1, and the edge padding is a constant 0 padding of 1 layer; the second layer convolution parameters: the number of convolution kernels is 256, the size is 1, and the step size is 1; the channel attention module consists of two fully connected layers with 16 and 256 hidden units respectively, and the spatial attention module consists of a two-dimensional convolution with 1 convolution kernel, 7x7 size, step size 1, and 3 layers of constant 0 padding of the edge; the second layer of two-dimensional convolution parameters: the number of convolution kernels is 256, the size is 3x3, the step size is 1, and the edge padding is a constant 0 padding of 1 layer; the first layer of two-dimensional convolution parameters of the 7th and 8th layers of MAFN-Block: the number of convolution kernels is 512, size 3x3, stride 2, edge padding is 1 layer of constant 0 padding; the temporal attention module layer consists of two one-dimensional convolutions: the first layer of convolution parameters: the number of convolution kernels is 128, the size is 3, the stride is 1, and the edge padding is 1 layer of constant 0 padding; the second layer of convolution parameters: the number of convolution kernels is 512, the size is 1, and the stride is 1; the channel attention module consists of two fully connected layers with 32 and 512 hidden units respectively, and the spatial attention module consists of a two-dimensional convolution with 1 convolution kernel, size 7x7, stride 1, and edge padding is 3 layers of constant 0 padding; the second layer of two-dimensional convolution parameters: the number of convolution kernels is 512, the size is 3x3, the stride is 1, and the edge padding is 1 layer of constant 0 padding;The parameters of the final fully connected layer are related to the dataset. ;

[0053] The timing module consists of a bidirectional long short-term memory network with a hidden layer dimension of 1024.

[0054] The main classifier and auxiliary classifier form a classification prediction module by sharing weights. The module consists of a fully connected layer, the input vector dimension is 1024, and the output vector dimension is related to the data set.

[0055] Furthermore, the training process of training the sign language recognition model through the training set includes:

[0056] Step 1: extract a video segment from the preprocessed training set, input it into the sign language recognition model, and obtain a first prediction result and a second prediction result obtained by the classification prediction module for predicting the probability of the sign language category based on the visual features and the context features respectively;

[0057] Step 2: Calculate the loss based on the first prediction result and the second prediction result to obtain the first loss result and the second loss result;

[0058] Step 3: Update the parameters of the sign language recognition model based on the first loss result and the second loss result to obtain an updated sign language recognition model;

[0059] Step 4: re-extract the video segments, repeat steps 1 to 3, and update the parameters of the updated sign language recognition model. If the model converges, the training is completed. If the model has not converged until all the video segments in the preprocessed training set are extracted, the order of the video segments in the preprocessed training set is disrupted, and steps 1 to 3 are repeated until the model converges.

[0060] Specifically, the model training process in this embodiment is to update the model parameters to minimize the loss caused by the deviation between the estimated distribution and the true distribution by continuously calling the adaptive moment estimation gradient descent algorithm. According to the objective function, the loss of the current model is calculated by a forward pass operation. Then, the update gradient of each parameter in the model is calculated based on the loss of the current model, and the parameters are updated through a backward pass operation, so that the data distribution fitted by the updated model approaches the real data distribution.

[0061] This example uses CTC and dynamic programming to obtain the probability distribution Decode the word sequence, first the "blank" label class and vocabulary The final expanded vocabulary was created Next, define a path The input feature sequence F is of length T, and the global parameter of the model is represented by θ. Then the conditional probability of a specific path π can be calculated as:

[0062]

[0063] The sentence l composed of sign language vocabulary decoded by CTC is the ground truth recognition result of continuous sign language events. The probability of successful recognition is the sum of the probabilities of all possible paths:

[0064]

[0065] The CTC loss function can be calculated as the negative logarithm of the probability of correctly labelling the sequence:

[0066]

[0067] This example adds a CTC loss after the visual features. and a knowledge distillation loss

[0068]

[0069] The high temperature parameter τ is used to "soften" the probability distribution of the spike response that CTC may produce. Z represents the feature sequence of the context feature after passing through the main classifier, and It can represent the feature sequence of visual features after passing through the classifier, and KL is the KL divergence calculation, also known as relative entropy.

[0070] The final loss function It can be expressed as:

[0071]

[0072] In the formula, For the final loss, is the main loss for CTC decoding, is the auxiliary loss for CTC decoding, is the knowledge distillation loss, μ is a hyperparameter, p is the recognition success probability of continuous sign language events, l is the actual recognition result of continuous sign language events, F is the feature sequence of visual features or context features, θ is the global parameter of the model, v is the auxiliary classifier, KL is the KL divergence calculation, Z is the feature sequence of context features after the classification prediction module, is the feature sequence of the visual features after the classification prediction module, and τ is the high temperature parameter.

[0073] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A continuous sign language recognition method based on spatiotemporal multi-attention mechanism fusion, characterized in that: include: Get the sign language video to be recognized; The sign language video to be recognized is input into a preset sign language recognition model, and a continuous sign language recognition result is output, wherein the sign language recognition model is obtained by training with a training set, the training set includes several sign language videos, and the sign language recognition model is constructed based on a neural network combined with a spatiotemporal multi-attention mechanism.

2. The continuous sign language recognition method based on spatiotemporal multi-attention mechanism fusion according to claim 1 is characterized in that: Before training the sign language recognition model through the training set, the method further includes preprocessing the sign language video in the training set, wherein the preprocessing includes: scaling the sign language video, randomly cropping the scaled sign language video, and randomly horizontally flipping the cropped sign language video to obtain a preliminarily processed sign language video; The sign language video after preliminary processing is randomly divided into video groups, and re-joined into preset sizes to obtain a final sign language video, thus completing the pre-processing.

3. The continuous sign language recognition method based on spatiotemporal multi-attention mechanism fusion according to claim 1 is characterized in that: The sign language recognition model includes: a feature extraction module, a timing module, and a classification prediction module, wherein the feature extraction module is used to extract the visual features of the sign language video to be recognized, and send them to the timing module and the classification prediction module respectively; the timing module is used to extract context features based on the visual features, and send them to the classification prediction module; the classification prediction module is used to predict the sign language category probability based on the visual features and the context features respectively, and output a continuous sign language recognition result.

4. The method for continuous sign language recognition based on spatiotemporal multi-attention mechanism fusion according to claim 3 is characterized in that: The feature extraction module includes: a first convolution unit, a channel attention unit, a spatial attention unit, a temporal attention unit, and a second convolution unit. The sign language video to be recognized first passes through the first convolution unit, and then passes through the first branch composed of the channel attention unit and the spatial attention unit and the second branch composed of the temporal attention unit, and finally passes through the second convolution unit to output the visual features.

5. The method for continuous sign language recognition based on spatiotemporal multi-attention mechanism fusion according to claim 3, characterized in that: The timing module uses a bidirectional long short-term memory network to extract context features based on the visual features; The classification prediction module predicts the probability of the sign language category based on the visual features and the context features in a manner where the main classifier and the auxiliary classifier share weights.

6. The method for continuous sign language recognition based on spatiotemporal multi-attention mechanism fusion according to claim 1, characterized in that: The training process of training the sign language recognition model by using the training set includes: Step 1: extract a video segment from the preprocessed training set, input it into the sign language recognition model, and obtain a first prediction result and a second prediction result obtained by the classification prediction module for predicting the probability of the sign language category based on the visual features and the context features respectively; Step 2: Calculate the loss based on the first prediction result and the second prediction result to obtain a first loss result and a second loss result; Step 3: updating the parameters of the sign language recognition model based on the first loss result and the second loss result to obtain an updated sign language recognition model; Step 4: re-extract the video segment, repeat steps 1 to 3, and update the parameters of the updated sign language recognition model. If the model converges, the training is completed.

7. The method for continuous sign language recognition based on spatiotemporal multi-attention mechanism fusion according to claim 6, characterized in that: The training process of training the sign language recognition model through the training set also includes: if the model has not converged until the video segments in the preprocessed training set are extracted, the order of the video segments in the preprocessed training set is disrupted, and steps 1 to 3 are repeated until the model converges.

8. The method for continuous sign language recognition based on spatiotemporal multi-attention mechanism fusion according to claim 6, characterized in that: The loss function for performing the loss calculation is: In the formula, For the final loss, is the main loss for CTC decoding, is the auxiliary loss for CTC decoding, is the knowledge distillation loss, μ is a hyperparameter, p is the recognition success probability of continuous sign language events, l is the actual recognition result of continuous sign language events, F is the feature sequence of visual features or context features, θ is the global parameter of the model, v is the auxiliary classifier, KL is the KL divergence calculation, z is the feature sequence of context features after the classification prediction module, is the feature sequence of the visual features after the classification prediction module, and τ is the high temperature parameter.

Images