Continuous Sign Language Recognition Method Based on Adaptive Multi-Scale Attention Temporal Network

Abstract

The present invention is a continuous sign language recognition method based on an adaptive multi-scale attention temporal network. The method includes: adjusting the image size of the input sign language image sequence and performing preprocessing; inputting the preprocessed sign language image sequence into the Resnet18 residual network for spatial feature extraction to obtain spatial features; inputting the spatial features into the adaptive multi-scale attention temporal network to extract multi-scale spatio-temporal features of the spatial features, and inputting the multi-scale spatio-temporal features into a two-layer tandem bidirectional long short-term memory network to extract spatio-temporal features of their long-term temporal connections; using a fully connected layer and a softmax activation function to convert the spatio-temporal features of the long-term temporal connections into a classification probability matrix; using the classification probability matrix for training and decoding to obtain the label of the sign language sequence. This method solves the problem of insufficient extraction of the temporal features of sign language words caused by using a fixed-time receptive field in previous work and improves the recognition performance of the network.

Description

Technical Field

[0001] The technical solution of the present invention relates to the fields of deep learning image processing and pattern recognition, and specifically to a continuous sign language recognition method based on an adaptive multi-scale attention temporal network. Background Art

[0002] The purpose of continuous sign language recognition is to translate a complete sign language sentence into a complete sentence. A complete sign language sentence is composed of several words. The task of continuous sign language recognition is to segment the complete sign language sentence into several sign language words, and then translate them into a complete sentence sequence through sentence synthesis. Continuous sign language recognition involves multiple research fields such as computer vision, human-computer interaction, pattern recognition, and natural language processing.

[0003] Sign language is an important body language and an important means of communication between deaf people and normal people. The birth of continuous sign language recognition technology has facilitated the normal communication between deaf people and normal people. Sign language recognition is generally divided into two categories: isolated sign language recognition and continuous sign language recognition. Continuous sign language recognition usually uses isolated sign language recognition as the basis, recognizes isolated sign language words, and then performs word splicing. Traditional continuous sign language recognition methods use manually crafted features for modeling. In recent years, deep learning has been applied to continuous sign language recognition, and there have been obvious improvements in problems such as gesture occlusion and large gesture variations in continuous sign language recognition, as well as some shortcomings of traditional sign language recognition. In existing sign language datasets, the label corresponding to a continuous sign language video is the label of the entire sentence, and the alignment between the frames in the video and the sign language words in the sign language label is unknown. Therefore, continuous sign language recognition is a weakly supervised problem. The CTC (Connectionist Temporal Classification) algorithm can solve the weakly supervised problem of video labels. It was originally proposed to solve the alignment problem in speech recognition. The CTC algorithm can automatically complete the alignment task between the input video sequence and the text sequence. Continuous sign language recognition involves spatial and temporal problems. The people and their body movements and positions in the image constitute spatial information, while the video composed of continuous images constitutes temporal information. Simply focusing only on spatial information or temporal information will ignore important information in sign language. Most previous work uses a combination of 3D-CNN or 2D-CNN + 1D-CNN to achieve feature extraction. Dongxu Li et al. used a combination of 3D-CNN for feature extraction in "TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation", modeled the video semantics locally and globally using multi-granularity temporal information, greatly alleviated the need for gesture segmentation, promoted local semantic consistency using a cross-scale attention model, and used a fixed temporal receptive field for sign language translation. However, in actual situations, due to different sign language words and even different sign language demonstrators, the time of each sign language word is different. Using a fixed temporal receptive field as the input will make the extraction of the time features of sign language words inaccurate.

[0004] CN113537024A discloses a continuous sign language recognition method based on an encoder-decoder. This method extracts pose information through a pre-trained convolutional neural network, uses a bidirectional gated recurrent network and a multi-layer residual stacked gated recurrent network for temporal encoding, fuses the low-level features and high-level semantics of sign language through a multi-layer attention fusion mechanism, and finally uses greedy decoding to obtain the result of sign language recognition. The disadvantages of this method are as follows: it ignores the short-term correlation between sign languages, models long-term sign language frames, but does not process the adjacent sign language frames, only processes single frames, and there is no correlation between two frames. CN112329690A discloses a sign language recognition method. In this method, 3D-CNN is used to extract spatio-temporal features. The input convolution kernel in 3D-CNN remains unchanged all the time, and it also has a fixed receptive field, does not pay attention to the short-term time length, and cannot accurately extract the time features of sign language words. Summary of the Invention

[0005] In view of the deficiencies of the prior art, the technical problem to be solved by the present invention is to provide a continuous sign language recognition method based on an adaptive multi-scale attention temporal network. This method obtains multi-scale spatio-temporal features with different temporal receptive fields by using one-dimensional convolution in the multi-scale time dimension, assigns attention coefficients to the multi-scale spatio-temporal features respectively by using attention, and fuses the spatio-temporal features of different receptive fields through average pooling to obtain multi-scale spatio-temporal features, solves the problem of insufficient extraction of the time features of sign language words caused by using one-dimensional convolution with a fixed temporal receptive field in previous work, enhances the robustness of the network, and improves the recognition performance of the network.

[0006] The technical solution adopted by the present invention to solve the above technical problem is as follows:

[0007] A continuous sign language recognition method based on an adaptive multi-scale attention temporal network, characterized in that the method includes the following contents:

[0008] Adjust the image size of the input sign language image sequence and perform preprocessing;

[0009] Input the preprocessed sign language image sequence into the Resnet18 residual network for spatial feature extraction to obtain spatial features;

[0010] Input the spatial features into the adaptive multi-scale attention temporal network to extract the multi-scale spatio-temporal features of the spatial features, input the multi-scale spatio-temporal features into a two-layer tandem bidirectional long short-term memory network to extract the spatio-temporal features of its long-term time connection; use a fully connected layer and a softmax activation function to convert the spatio-temporal features of the long-term time connection into a classification probability matrix;

[0011] Construct a connection temporal classification loss function through the classification probability matrix to train the neural network;

[0012] Decode through the trained neural network to obtain the labels of the sign language sequence;

[0013] The adaptive multi-scale attention temporal network includes a multi-scale temporal convolution module, a channel attention module, and an adaptive multi-scale attention module. The multi-scale temporal convolution module uses one-dimensional convolutions with different kernel sizes to obtain multi-scale spatio-temporal features, and stacks and combines the multi-scale spatio-temporal features to generate combined spatio-temporal features. The input feature dimension of the multi-scale temporal convolution module is [T, C], and the output feature dimension is [T, C, N], where T is the number of frame sequences, C is the number of channels, and N is the number of scale types;

[0014] The input of the channel attention module is the combined spatio-temporal features. Use max pooling and average pooling to obtain two channel information features of the combined spatio-temporal features in the scale dimension respectively. The two channel information features are respectively processed by two-dimensional convolutions with a kernel size of [1, 1], the Relu activation function, and two-dimensional convolutions with a kernel size of [1, 1] in sequence, and then the two processed channel information features are element-wise added and processed by the Sigmoid activation function to obtain the channel attention coefficient. The channel attention coefficient and the combined spatio-temporal features are element-wise multiplied and processed by the reshape function to obtain the combined spatio-temporal features enhanced by channel attention;

[0015] The input of the adaptive multi-scale attention module is the combined spatio-temporal features enhanced by channel attention. The combined spatio-temporal features enhanced by channel attention are respectively processed by max pooling and average pooling in the channel dimension to obtain two multi-scale information features. The two multi-scale information features are concatenated along the channel direction, and an adaptive multi-scale attention coefficient is generated through two-dimensional convolution and the Sigmoid activation function. The combined spatio-temporal features enhanced by channel attention and the adaptive multi-scale attention coefficient are element-wise multiplied, and processed by the reshape function to obtain the combined spatio-temporal features enhanced by adaptive multi-scale attention. Then, through two-dimensional average pooling and one-dimensional max pooling, the multi-scale spatio-temporal feature L is obtained.

[0016] The specific steps are as follows:

[0017] In the first step, input the image sequence V and perform preprocessing:

[0018] Input the image sequence V=(v1,...,v i ,...,v T ), where T is the number of frame sequences of the input image sequence V, which respectively represent the first frame,..., the i-th frame,..., the T-th frame of the original sign language image sequence. The input video size is 210×260, which is converted to 256×256, and then through preprocessing to obtain the image sequence W. The preprocessing process includes random cropping, random horizontal flipping, random scaling, and normalization to obtain the preprocessed sign language image sequence

[0019] Step 2: Extract spatial features from the preprocessed image sequence W:

[0020] Input the image sequence W obtained by preprocessing in the first step into the residual network Resnet18 to extract spatial features, and obtain the spatial feature The dimension of F is (1, T, 1024), where the second dimension is the time dimension, the third dimension is the channel dimension, and the first dimension is a placeholder dimension to meet the requirements of the residual network Resnet18, without actual meaning;

[0021] Step 3: Extract multi-scale spatio-temporal features of the spatial feature F:

[0022] Input the spatial feature F obtained in the second step into the adaptive multi-scale attention temporal network. The adaptive multi-scale attention temporal network consists of a multi-scale temporal convolution module, a channel attention module, and an adaptive multi-scale attention module. The specific operation of extracting the multi-scale spatio-temporal features of the spatial feature F is as follows:

[0023] Input the spatial feature F obtained in the second step into the multi-scale temporal convolution module. The multi-scale temporal convolution module includes multiple parallel one-dimensional convolutions with different kernel sizes, stacking operations, and reshape functions. The specific structure is Conv1D_3, Conv1D_5, Conv1D_7, Conv1D_9, Conv1D_11, which respectively represent one-dimensional convolutions with kernel sizes of 3, 5, 7, 9, 11 and padding sizes of 1, 2, 3, 4, 5. After the spatial feature F passes through one-dimensional convolutions of five different scales, spatio-temporal features I3, I5, I7, I9, I 11 , their dimensions are all (1, 1024, T); combine the spatio-temporal features of different receptive fields obtained and then obtain the combined spatio-temporal feature I′ through the reshape function, and its feature dimension is (T, 1024, 5, 1). At this time, 1 is the placeholder dimension;

[0024] Input the obtained combined spatio-temporal feature I′ into the channel attention module to enhance channel attention. The channel attention module includes two parallel average pooling and max pooling. The outputs of average pooling and max pooling are respectively connected to two-dimensional convolutions with kernel sizes of [1, 1], Relu activation functions, and two-dimensional convolutions with kernel sizes of [1, 1]; after element-wise addition and Sigmoid activation function processing, obtain the channel attention coefficient, and then perform Hadamard product with the input of the channel attention module and obtain the combined spatio-temporal feature I″ with enhanced channel attention through the reshape function; the specific process is:

[0025] First, two-channel information features G are screened out through max pooling and average pooling respectively max and G avg , their dimensions are (T, 1024, 1, 1), expressed as shown in the following formulas (9) and (10),

[0026] G max = MaxPool(I′) (9)

[0027] G avg = AvgPool(I′) (10)

[0028] Among them, MaxPool(·) represents the max pooling function, and AvgPool(·) represents the average pooling function,

[0029] The two-channel information features G max and G avg are respectively passed through a two-dimensional convolution with a kernel size of [1, 1] to reduce the channel dimension, then activated through the Relu activation function respectively, and then passed through a two-dimensional convolution with a kernel size of [1, 1] to increase the channel dimension to obtain G′ max and G′ avg , expressed as shown in the following formulas (11) and (12),

[0030] G′ max = Conv2D 1×1 (Relu(Conv2D 1×1 (G max ))) (11)

[0031] G′ avg = Conv2D 1×1 (Relu(Conv2D 1×1 (G avg ))) (12)

[0032] Among them, Conv2D 1×1 (·) represents a two-dimensional convolution with a kernel size of [1, 1], and Relu(·) represents the Relu activation function,

[0033] Finally, the corresponding elements of G′ max and G′ avg obtained in formulas (11) and (12) are added together, passed through the Sigmoid activation function to generate channel attention coefficients. After the channel attention coefficients and the combined spatio-temporal feature I′ are subjected to the Hadamard product, they are then transformed in dimension through the reshape function to obtain the combined spatio-temporal feature I″ with enhanced channel attention, and its feature dimension is (1, 1024, 5, T), expressed as shown in the following formula (13),

[0034] I″ = reshape(I′ ⊙ (Sigmoid(G′ max + G′ avg ))) (13)

[0035] In formula (13), + represents the element-wise addition of two matrices, ⊙ represents the Hadamard product, Sigmoid(·) represents the Sigmoid activation function, and reshape(·) represents the function for converting the dimension of the feature vector;

[0036] The combined spatio-temporal feature I″ with enhanced channel attention obtained from formula (13) is input into the adaptive multi-scale attention module. The adaptive multi-scale attention module includes two parallel average pooling and max pooling. The outputs of the two are processed through a channel dimension concatenation operation, a two-dimensional convolution with a convolutional kernel size of [5, 3] and a padding size of [2, 1], and a Sigmoid activation function to output the adaptive multi-scale attention coefficient Att mul The adaptive multi-scale attention coefficient Att mul and the input of the adaptive multi-scale attention module are subjected to the Hadamard product, and then the adaptive multi-scale attention enhanced combined spatio-temporal feature I″′ is obtained through the reshape function. Finally, the multi-scale spatio-temporal feature L is obtained through two-dimensional average pooling and one-dimensional max pooling operations in sequence;

[0037] The specific process is as follows:

[0038] First, two multi-scale information features R max and R avg are respectively selected through max pooling and average pooling in the channel dimension. Their feature dimensions are both (1, 1, 5, T), expressed as formulas (14) and (15),

[0039] R max = MaxPool_c(I″) (14)

[0040] R ayg = AvgPool_c(I″) (15)

[0041] In formulas (14) and (15), MaxPool_c(·) represents the max pooling function in the channel dimension, i.e., the second dimension, and AvgPool_c(·) represents the average pooling function in the channel dimension, i.e., the second dimension,

[0042] Then, the two multi-scale information features R max and R avg are concatenated according to the channel dimension, and then a convolution operation is performed through a two-dimensional convolution with a convolutional kernel size of [5, 3] and a padding size of [2, 1]. Then, the Sigmoid activation function is used to obtain the adaptive multi-scale attention coefficient Att mul, with its feature dimension being (1, 1024, 5, T), expressed as formula (16),

[0043] Att mul = Sigmoid(Conv2D 5×3 (contact(R max , R avg ))) (16)

[0044] In formula (16), contact(·) represents a function that concatenates feature vectors along the channel dimension, Conv2D 5×3 (·) represents a function of two-dimensional convolution with a convolutional kernel size of [5, 3] and a padding size of [2, 1], and Sigmoid(·) represents the Sigmoid activation function.

[0045] Finally, the adaptive multi-scale attention coefficient Att mul obtained in formula (16) and the combined spatio-temporal feature I″ with channel attention enhancement are subjected to the Hadamard product, and then the adaptive multi-scale attention-enhanced combined spatio-temporal feature I′″ is obtained through the reshape function, with its feature dimension being (T, 1024, 5, 1), expressed as shown in the following formula (17):

[0046] I″′ = reshape(I″⊙Att mul ) (17)

[0047] In formula (16), ⊙ represents the Hadamard product, and reshape(·) represents a function that transforms the dimension of the feature vector.

[0048] The adaptive multi-scale attention-enhanced combined spatio-temporal feature I′″ obtained in formula (17) is subjected to two-dimensional average pooling and one-dimensional max pooling to obtain the multi-scale spatio-temporal feature with its feature dimension being (T′, 1024), where T′ = T / 2, and it is expressed as shown in the following formula (18):

[0049]

[0050] In formula (18), AvgPool 2D (·) represents the two-dimensional average pooling function, and MaxPool 1D represents the one-dimensional max pooling function.

[0051] Fourth step, extract the spatio-temporal feature of the long-term time connection of the multi-scale spatio-temporal feature L:

[0052] Input the multi-scale spatio-temporal feature L obtained in the above third step into a two-layer tandem bidirectional long short-term memory network to extract its spatio-temporal feature of long-term time connection

[0053] Step 5: Obtain the classification probability matrix according to the spatio-temporal feature E of the long-term time connection:

[0054] Input the spatio-temporal feature E of the long-term time connection obtained in the fourth step into a fully connected layer, and then activate it through the Softmax activation function to obtain the classification probability matrix Its feature dimension is (T′, 1296). Each vector in Y is a probability value, and the sum of the probability values in the vector direction is 1.

[0055] Step 6: Construct the connectionist temporal classification loss function Train the neural network:

[0056] Input the classification probability matrix obtained in the fifth step and the label l of the data set into the connectionist temporal classification network (CTC). The connectionist temporal classification network introduces a blank label <blank>As an auxiliary marker, the predicted probability is obtained by summing the conditional probabilities of all possible paths, and the input is the classification probability matrix After that, the predicted probability P(l|Y) is obtained, and its expression is shown in the following formula (21),

[0057]

[0058] In formula (21), Π = (π1,..., π i ,..., π s ) represents the set of all possible paths, π i represents the i-th possible path, s represents the total number of possible paths, B is a many-to-one mapping used to remove empty labels and duplicate labels in the path Π, and B -1 is the inverse operation of B,

[0059] The purpose of the connectionist temporal classification network is to maximize the predicted probability. Therefore, it uses the negative of the logarithm of the predicted probability as the loss function, and its expression is shown in the following formula (22),

[0060]

[0061] In formula (22), log(·) represents the natural logarithm function with base e;

[0062] Step 7: Use the trained neural network to predict the sign language sequence and output the label H of the sign language sequence;

[0063] Through the neural network trained in the sixth step, the beam search decoding (Beam) is used to decode the classification probability matrix Y obtained in the fifth step to obtain the label H of the sign language sequence, and its expression is shown in the following formula (23),

[0064] H = Beam(Y) (23)

[0065] In formula (23), Beam(·) represents the beam search decoding function.

[0066] The neural network described above includes the residual network Resnet18, the adaptive multi-scale attention temporal network, the two-layer bidirectional long short-term memory network connected in series, and the connectionist temporal classification network. The residual network Resnet18, the adaptive multi-scale attention temporal network, and the two-layer bidirectional long short-term memory network connected in series are connected in sequence. The output of the two-layer bidirectional long short-term memory network connected in series is processed by a fully connected layer and a Softmax activation function and then connected to the connectionist temporal classification network. The input of the connectionist temporal classification network is the classification probability matrix processed by the fully connected layer and the Softmax activation function and the label l of the dataset.

[0067] Compared with the prior art, the beneficial effects of the present invention are:

[0068] The prominent substantive features of the present invention are as follows:

[0069] (1) In the method of the present invention, the adaptive multi-scale attention temporal network is a novel multi-scale temporal network, which can overcome the problem of different lengths of sign language words in actual situations and avoid the problem of mismatch between the feature blocks and the actual length of sign language words caused by using a temporal network with a fixed time receptive field. Most previous works on continuous sign language recognition used a fixed time receptive field to extract short-term spatio-temporal features. However, in actual situations, due to different sign language words and different sign language demonstrators, sign language words often have different temporal lengths. The adaptive multi-scale attention temporal network can obtain spatio-temporal features with different receptive fields through convolutions of different sizes, and assign weights to the spatio-temporal features of different receptive fields through attention, that is, after generating features with different receptive fields, channel attention and adaptive multi-scale attention processing are performed, and then the spatio-temporal features of different receptive fields are fused through average pooling, so as to extract multi-scale spatio-temporal features.

[0070] (2) The present invention uses the attention mechanism to assign different weights to features of different scales, and then through the fusion of features, multi-scale is achieved, and the importance of different features can be adaptively distinguished. In addition, this method can also be migrated to other sequence learning models to improve the processing performance of time multi-scale.

[0071] The remarkable progress of the present invention is:

[0072] (1) Compared with CN112329690A, the method of the present invention can capture spatio-temporal features of a time receptive field close to the actual length of sign language words through multiple one-dimensional convolutions and max pooling, overcoming the problem that a fixed time receptive field cannot well extract the temporal features of each sign language word, and improving the recognition effect of the network.

[0073] (2) Compared with CN113537024A, the advantages of the method of the present invention are that one-dimensional convolution is added to extract the temporal relationship between adjacent frames, enhancing the short-term spatio-temporal correlation between features, so that the recognition of the entire network for sign language words is not limited to the frame level, but focuses on a short-term sign language action. The short term refers to a continuous number of frames during the sign language action process. The spatio-temporal features are extracted through one-dimensional convolutions with different convolution kernel sizes, which can fully strengthen the temporal correlation before and after the gesture and improve the recognition effect.

[0074] (3) The method of the present invention uses multi-scale one-dimensional convolutions to capture spatio-temporal features of different time receptive fields, and then different weights are assigned to the spatio-temporal features of different time receptive fields through the attention mechanism to achieve the selection of their importance, so that the temporal features of sign language words can be fully extracted and the performance of sign language recognition can be enhanced.

[0075] (4) The method of the present invention splices multi-scale spatio-temporal features according to a new dimension, enhances the channel attention of the spliced features, then extracts the information in the channel direction of the channel attention-enhanced features through average pooling and max pooling respectively, and performs multi-scale attention operations on the combined spatio-temporal features with enhanced channel attention through two-dimensional convolution and Sigmoid activation function, and finally obtains the final multi-scale features through multi-scale attention.

[0076] (5) The method of the present invention uses a combination of two-dimensional convolution and one-dimensional convolution to replace three-dimensional convolution, which can effectively reduce the number of parameters of the neural network, speed up the calculation speed of the neural network, and can more concisely control the temporal receptive field of spatio-temporal features according to the size of the one-dimensional convolution kernel, with fewer parameters compared to three-dimensional convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The present invention will be further described below in conjunction with the drawings and embodiments.

[0078] Figure 1 is a flow schematic block diagram of the continuous sign language recognition method based on the adaptive multi-scale attention time series network of the present invention.

[0079] Figure 2 is a structural schematic diagram of the adaptive multi-scale attention time series network.

[0080] Figure 3 is a structural schematic diagram of the multi-scale time convolution module.

[0081] Figure 4 is a structural schematic diagram of the channel attention module.

[0082] Figure 5 is a structural schematic diagram of the adaptive multi-scale attention module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0083] The short-term relevance of sign language means that sign language is a continuous action, and there is a motion relationship between adjacent frames before and after. Therefore, there must be a connection between adjacent or nearby frames. The short-term relevance is to focus on the relationship between adjacent or nearby frames.

[0084] The receptive field refers to the area where the calculation of a certain element in the feature is affected by the input feature. For example, the temporal receptive field of the feature obtained by using a convolution with a convolution kernel size of 3 is 3. The multi-scale receptive field in this application is realized through one-dimensional convolution with different convolution sizes. The features generated by one-dimensional convolution with different convolution kernel sizes have different temporal receptive fields, and finally the features with different receptive fields are effectively fused, which is the fusion of features with multiple different receptive fields.

[0085] The adaptive multi-scale attention temporal network of the present invention uses one-dimensional convolutions with different kernel sizes to extract temporal features, and realizes the adaptive adjustment of the importance of different temporal features through attention.

[0086] Figure 1 The illustrated embodiment shows that the process of the continuous sign language recognition method of the adaptive multi-scale attention temporal network of the present invention is as follows:

[0087] Preprocess the input image sequence → Extract the spatial features of the image sequence → Extract the multi-scale spatio-temporal features of the spatial features → Extract the spatio-temporal features of the long-term temporal connection of the multi-scale spatio-temporal features → Obtain the classification probability matrix → Construct a connection temporal classification loss function based on the classification probability matrix and train the neural network → Perform sign language prediction according to the trained neural network.

[0088] Figure 2 The illustrated embodiment represents an adaptive multi-scale attention temporal network, including a multi-scale temporal convolution module, a channel attention module, and an adaptive multi-scale attention module, where F is the spatial feature extracted from the image sequence. The specific content of the adaptive multi-scale attention temporal network is as follows: The input feature dimension is [T, 1024], which means T features of 1024 dimensions, where T is the time length (number of frame sequences) and 1024 is the number of channels. In this embodiment, one-dimensional convolutions with five different kernel sizes are used to extract temporal features to obtain 5 features of dimension [T, 1024], but these 5 features have different temporal receptive fields. After concatenating the 5 features with different temporal receptive fields, the dimension becomes [T, 1024, 5]. First, channel attention is used for channel enhancement, and then adaptive multi-scale attention is used for enhancement. The final feature dimension obtained is [T, 1024, 5]. Through two-dimensional average pooling, its feature dimension is transformed into [T, 1024]. Through two-dimensional average pooling, feature fusion of different receptive fields is realized. Finally, through one-dimensional max pooling, its feature dimension is transformed into [T / 2, 1024], reducing the time length for subsequent processing.

[0089] Figure 3 The illustrated embodiment represents a multi-scale temporal convolution module. The one-dimensional convolutions with different kernel sizes refer to five one-dimensional convolutions with kernel sizes of 3, 5, 7, 9, 11 and padding sizes of 1, 2, 3, 4, 5 respectively. They respectively perform convolution operations on the input F to generate five spatio-temporal features I3, I5, I7, I9, I 11 , and stack I3, I5, I7, I9, I 11 together according to the new dimension to obtain the combined spatio-temporal feature I′. The combined spatio-temporal feature I′ has one more dimension than the spatio-temporal features I3, I5, 17, I9, I 11 , that is, the scale type number dimension.

[0090] Figure 4 The illustrated embodiment represents a channel attention module. The input combined spatio-temporal feature I′ generates channel information features G respectively through multi-scale directional average pooling and multi-scale directional maximum pooling. avg and G max Then, through two-dimensional convolution, Relu activation function and two-dimensional convolution, G′ is generated. avg and G′ max The kernel size of the two-dimensional convolution is [1, 1]. G′ avg and G′ max are added element-wise (element-wise addition means adding according to the corresponding values of the two features. For example, for two 3×3 features added element-wise, it is the first value in the first row of the first feature added to the first value in the first row of the second feature). After activation through the Sigmoid activation function, the channel attention coefficient is obtained. Then, the Hadamard product is performed with the input of the channel attention module to obtain the combined spatio-temporal feature I″ with enhanced channel attention, which is the output of the channel attention module.

[0091] Figure 5 The illustrated embodiment represents an adaptive multi-scale attention module. The input combined spatio-temporal feature I″ with enhanced channel attention generates multi-scale information features R respectively through channel-wise average pooling and channel-wise maximum pooling. avg and R max After concatenating them along the channel dimension, through two-dimensional convolution and Sigmoid activation function, the adaptive multi-scale attention coefficient Att mul is generated. The kernel size of the two-dimensional convolution is [5, 3]. The adaptive multi-scale attention coefficient Att mul and the combined spatio-temporal feature I″ with enhanced channel attention perform the Hadamard product to generate the combined spatio-temporal feature I″′ with enhanced adaptive multi-scale attention. Through two-dimensional average pooling and one-dimensional maximum pooling respectively, the multi-scale spatio-temporal feature L is obtained. The kernel of the two-dimensional average pooling is [5, 1], and the kernel of the one-dimensional maximum pooling is 2.

[0092] Embodiment 1

[0093] The continuous sign language recognition method based on the adaptive multi-scale attention time series network in this embodiment is as follows:

[0094] In the first step, the input image sequence V is preprocessed:

[0095] The input image sequence V = (v1,..., v i ,...v T ), where \(T\) is the number of frames in the input image sequence \(V\), which represent the first frame, …, the \(i\)-th frame, …, the \(T\)-th frame of the original sign language image sequence respectively. The size of the input video is \(210\times260\), which is converted to \(256\times256\), and then the image sequence \(W\) is obtained through preprocessing. The preprocessing process includes random cropping, random horizontal flipping, random scaling, and normalization, which is expressed as shown in the following formula (1).

[0096]

[0097] In formula (1), \(Resize(·)\) is a function for resizing the image, \(RandomCrop(·)\) is a function for randomly cropping the image, \(RandomFlip(·)\) is a function for randomly horizontally flipping the image, \(TemporalRescale(·)\) is a function for randomly temporally scaling the image sequence, and \(Normalize(·)\) is a normalization function for changing the pixel values of the image to the range \((0, 1)\); \(w\) t is the \(t\)-th frame image.

[0098] In the second step, spatial features are extracted from the preprocessed image sequence \(W\):

[0099] The image sequence \(W\) obtained through preprocessing in the first step is input into the ResNet18 residual network to extract spatial features, and the spatial features \(F\) has a dimension of \((1, T, 1024)\), where the second dimension is the time dimension and the third dimension is the channel dimension. The extraction of spatial features is expressed as shown in the following formula (2).

[0100]

[0101] In formula (2), \(Conv\) resnet18 (·) represents the function of the ResNet18 network, and \(f\) t represents the feature vector at time \(t\) in the spatial feature \(F\).

[0102] In the third step, multi-scale spatio-temporal features of the spatial feature \(F\) are extracted:

[0103] The spatial feature \(F\) obtained in the second step is input into the adaptive multi-scale attention temporal network, which is composed of a multi-scale temporal convolution module, a channel attention module, and an adaptive multi-scale attention module. The specific operation for extracting the multi-scale spatio-temporal features of the spatial feature \(F\) is as follows:

[0104] Input the spatial feature F obtained in the second step into the multi-scale temporal convolution module. The multi-scale temporal convolution module includes multiple parallel one-dimensional convolutions. The specific structure is Conv1D_3, Conv1D_5, Conv1D_7, Conv1D_9, Conv1D_11, which respectively represent one-dimensional convolutions with a kernel size of 3, 5, 7, 9, 11 and a padding size of 1, 2, 3, 4, 5. After the spatial feature F passes through one-dimensional convolutions of five different scales, spatio-temporal features I3, I5, I7, I9, I 11 are obtained. Their dimensions are all (1, 1024, T). The spatial feature F passing through one-dimensional convolutions of five different scales is expressed as shown in the following formulas (3) - (7).

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] Formulas (3)(4)(5)(6)(7). respectively represent one-dimensional convolutions with a kernel size of 3, 5, 7, 9, 11 and a padding size of 1, 2, 3, 4, 5. BN(·) represents the batch normalization function, whose full English name is Batch Normalization, which is a well-known algorithm.

[0111] Combine the spatio-temporal features with different receptive fields obtained from formulas (3)(4)(5)(6)(7) and then obtain the combined spatio-temporal feature I′ through the reshape function. The feature dimension of the combined spatio-temporal feature I′ is (T, 1024, 5, 1), which is expressed as shown in the following formula (8).

[0112] I′ = reshape(stack(I3, I5, I7, I9, I 11 )) (8)

[0113] In formula (8), stack(·) represents a function that stacks multiple feature vectors together according to a new dimension, and reshape(·) represents a function that transforms the dimension of the feature vector.

[0114] Input the combined spatio-temporal feature I′ obtained in formula (8) into the channel attention module for channel attention enhancement. First, screen out two channel information features G max and G avg respectively through multi-scale directional maximum pooling and multi-scale directional average pooling (at this time, the dimension of I′ is [T, 1024, 5, 1], where T represents the time direction and represents the time length; 1024 represents the channel direction; 5 represents the multi-scale direction and represents the direction of multi-scale feature stacking). Their dimensions are (T, 1024, 1, 1), and are expressed as shown in the following formulas (9) and (10).

[0115] G max = MaxPool(I′) (9)

[0116] G avg = AvgPool(I′) (10)

[0117] In formulas (9) and (10), MaxPool(·) represents the maximum pooling function, and AvgPool(·) represents the average pooling function.

[0118] Reduce the channel dimension of G max and G avg obtained in formulas (9) and (10) respectively through two-dimensional convolution with a kernel size of [1, 1], then activate them respectively through the Relu activation function, and then increase the channel dimension respectively through two-dimensional convolution with a kernel size of [1, 1] to obtain G′ max and G′ avg , which are expressed as shown in the following formulas (11) and (12).

[0119] G′ max = Conv2D 1×1 (Relu(Conv2D 1×1 (G max ))) (11)

[0120] G′ avg = Conv2D 1×1 (Relu(Conv2D 1×1 (G avg ))) (12)

[0121] In formulas (11) and (12), Conv2D 1×1 (·) represents two-dimensional convolution with a kernel size of [1, 1], and Relu(·) represents the Relu activation function.

[0122] Finally, G′ max and G′ avg The corresponding elements of the two are added, and the channel attention coefficient is generated through the Sigmoid activation function. After the channel attention coefficient and the combined spatio-temporal feature I′ are subjected to the Hadamard product, the dimension is then transformed through the reshape function to obtain the combined spatio-temporal feature I″ with enhanced channel attention, whose feature dimension is (1, 1024, 5, T), expressed as shown in the following formula (13).

[0123] I″ = reshape(I′ ⊙ (Sigmoid(G′ max + G′ avg ))) (13)

[0124] In formula (13), + represents the addition of corresponding elements of two matrices, ⊙ represents the Hadamard product, Sigmoid(·) represents the Sigmoid activation function, and reshape(·) represents the function for transforming the dimension of the feature vector.

[0125] The combined spatio-temporal feature I″ with enhanced channel attention obtained in formula (13) is input into the adaptive multi-scale attention module. First, two multi-scale information features R max and R avg are respectively screened out through max-pooling and average-pooling in the channel dimension. Their feature dimensions are both (1, 1, 5, T), expressed as shown in the following formulas (14) and (15).

[0126] R max = MaxPool_c(I″) (14)

[0127] R avg = AvgPool_c(I″) (15)

[0128] In formulas (14) and (15), MaxPool_c(·) represents the max-pooling function in the channel dimension, i.e., the second dimension, and AvgPool_c(·) represents the average-pooling function in the channel dimension, i.e., the second dimension.

[0129] Then, the two multi-scale information features R max and R avg are concatenated according to the channel dimension, and then convolution operation is performed through a two-dimensional convolution with a convolution kernel size of [5, 3] and a padding size of [2, 1]. Then, the Sigmoid activation function is used to obtain the adaptive multi-scale attention coefficient Att mul , whose feature dimension is (1, 1024, 5, T), expressed as shown in the following formula (16).

[0130] Att mul = Sigmoid(Conv2D 5×3 (contact(R max , R avg ))) (16)

[0131] In formula (16), contact(·) represents a function that concatenates feature vectors along the channel dimension, and Conv2D 5×3 (·) represents a function of two-dimensional convolution with a convolutional kernel size of [5, 3] and a padding size of [2, 1], and Sigmoid(·) represents the Sigmoid activation function.

[0132] Finally, the adaptive multi-scale attention coefficient Att obtained in formula (16) mul and the combined spatio-temporal feature I″ with enhanced channel attention are subjected to Hadamard product, and then the adaptive multi-scale attention enhanced combined spatio-temporal feature I″′ is obtained through the reshape function, and its feature dimension is (T, 1024, 5, 1), which is expressed as shown in the following formula (17).

[0133] I″′ = reshape(I″⊙Att mul ) (17)

[0134] In formula (16), ⊙ represents the Hadamard product, and reshape(·) represents a function that transforms the dimension of the feature vector.

[0135] The combined spatio-temporal feature I″′ with adaptive multi-scale attention enhancement obtained in formula (17) is subjected to two-dimensional average pooling and one-dimensional max pooling to obtain the multi-scale spatio-temporal feature L, and its feature dimension is (T′, 1024), where T′ = T / 2, and it is expressed as shown in the following formula (18).

[0136]

[0137] In formula (18), AvgPool 2D (·) represents the two-dimensional average pooling function, which reduces the dimension in the multi-scale direction, and MaxPool 1D represents the one-dimensional max pooling function.

[0138] Fourth step, extract the spatio-temporal feature of the long-term temporal connection of the multi-scale spatio-temporal feature L:

[0139] Input the multi-scale spatio-temporal feature L obtained in the above third step into a two-layer concatenated bidirectional long short-term memory network to extract its spatio-temporal feature of long-term temporal connection It is expressed as shown in the following formula (19).

[0140]

[0141] In formula (19), BLSTM(·) represents the function of a two-layer tandem bidirectional long short-term memory network. The bidirectional long short-term memory network is abbreviated as BLSTM and is fully called Bi-directional Long Short-Term Memory, which is a well-known algorithm;

[0142] Step 5: Obtain the classification probability matrix based on the spatio-temporal feature E of the long-term temporal connection:

[0143] Input the spatio-temporal feature E of the long-term temporal connection obtained in the fourth step into a fully connected layer, and then activate it through the Softmax activation function to obtain the classification probability matrix Its feature dimension is (T′, 1296). Each vector in Y is a probability value, and the sum of the probability values in the vector direction is 1. Its expression is shown in the following formula (20),

[0144]

[0145] In formula (20), W fc is the parameter matrix of the fully connected layer, b fc is the bias value, and softmax(·) is the Softmax activation function;

[0146] Step 6: Construct a connectionist temporal classification loss function Train the neural network:

[0147] Input the classification probability matrix obtained in the fifth step and the label l of the data set into the connectionist temporal classification network (Connectionist Temporal Classification, CTC). The connectionist temporal classification network introduces a blank label <blank>As an auxiliary marker, the predicted probability is obtained by summing the conditional probabilities of all possible paths, and the input is the classification probability matrix After that, the predicted probability P(l|Y) is obtained, and its expression is shown in the following formula (21),

[0148]

[0149] In formula (21), Π = (π1,..., π i ,..., π s ) represents the set of all possible paths, π i represents the i-th possible path, s represents the total number of possible paths, B is a many-to-one mapping used to remove empty labels and duplicate labels in the path Π, and B -1 is the inverse operation of B;

[0150] The purpose of the connection time series classification network is to maximize the predicted probability. Therefore, it uses the negative of the logarithm of the predicted probability as the loss function Its expression is shown in the following formula (22),

[0151]

[0152] In formula (22), log(·) represents the logarithmic function with the natural number e as the base;

[0153] Step 7: Use the trained neural network to predict the sign language sequence and output the label H of the sign language sequence;

[0154] Through the neural network trained in the sixth step, the classification probability matrix Y obtained in the fifth step is decoded by beam search decoding (Beam) to obtain the label H of the sign language sequence, and its expression is shown in the following formula (23),

[0155] H = Beam(Y) (23)

[0156] In formula (23), Beam(·) represents the beam search decoding function;

[0157] So far, the continuous sign language recognition based on the adaptive multi-scale attention time series network is completed.

[0158] The neural network described in the present invention includes an end-to-end entire network composed of a residual network Resnet18, an adaptive multi-scale attention time series network, a BLSTM, and a CTC.

[0159] The above continuous sign language recognition using an adaptive multi-scale attention temporal network, in which the bi-directional long short-term memory network is abbreviated as BLSTM, and its full name is Bi-directional Long Short-Term Memory. The connectionist temporal classification network is abbreviated as CTC, and its full name is Connectionist Temporal Classification. Batch normalization is abbreviated as BN, and its full name is Batch Normalization. The 18-layer residual network is abbreviated as Resnet18, and its full name is Residual Network 18, which are algorithms well-known in the technical field. The hardware used is a computer, and the graphics card model is RTX3090.

[0160] Where the present invention is not described shall be applicable to the prior art.< / blank> < / blank>

Claims

1. A continuous sign language recognition method based on an adaptive multi-scale attention time series network, characterized in that, The method includes the following steps: Resize and preprocess the input sign language image sequence; Input the preprocessed sign language image sequence into the Resnet18 residual network for spatial feature extraction to obtain spatial features; Input the spatial features into the adaptive multi-scale attention temporal network to extract the multi-scale spatio-temporal features of the spatial features, input the multi-scale spatio-temporal features into the two-layer concatenated bidirectional long short-term memory network to extract the spatio-temporal features of their long-term temporal connections; use the fully connected layer and the softmax activation function to convert the spatio-temporal features of the long-term temporal connections into a classification probability matrix; Use the classification probability matrix for training decoding to obtain the label of the sign language sequence; The adaptive multi-scale attention temporal network includes a multi-scale temporal convolution module, a channel attention module, and an adaptive multi-scale attention module. The multi-scale temporal convolution module uses one-dimensional convolutions with different kernel sizes to obtain multi-scale spatio-temporal features, and stacks and combines the multi-scale spatio-temporal features together to generate combined spatio-temporal features; the input feature dimension of the multi-scale temporal convolution module is [T, C], and the output feature dimension is [T, C, N], where T is the number of frame sequences, C is the number of channels, and N is the number of scale types; The input of the channel attention module is the combined spatio-temporal features. Use max pooling and average pooling to respectively obtain two channel information features of the combined spatio-temporal features in the scale dimension. The two channel information features are respectively processed by two-dimensional convolutions with a kernel size of [1, 1], the Relu activation function, and two-dimensional convolutions with a kernel size of [1, 1]. Then, the two processed channel information features are added element-wise and processed by the Sigmoid activation function to obtain the channel attention coefficient. The channel attention coefficient and the combined spatio-temporal features are multiplied element-wise and then obtained through the reshape function to get the combined spatio-temporal features enhanced by channel attention; The input of the adaptive multi-scale attention module is the combined spatio-temporal features enhanced by channel attention. The combined spatio-temporal features enhanced by channel attention are respectively processed by max pooling and average pooling in the channel dimension to obtain two multi-scale information features. The two multi-scale information features are concatenated along the channel direction, and an adaptive multi-scale attention coefficient is generated through two-dimensional convolution and the Sigmoid activation function. The combined spatio-temporal features enhanced by channel attention and the adaptive multi-scale attention coefficient are multiplied element-wise, and then obtained through the reshape function to get the combined spatio-temporal features enhanced by adaptive multi-scale attention. After that, through two-dimensional average pooling and one-dimensional max pooling, the multi-scale spatio-temporal feature L is obtained.

2. The continuous sign language recognition method based on the adaptive multi-scale attention time series network according to claim 1, wherein The residual network Resnet18, the adaptive multi-scale attention time series network, and the two-layer bidirectional long short-term memory network connected in series constitute a neural network. The neural network also includes a connection time series classification network. The residual network Resnet18, the adaptive multi-scale attention time series network, and the two-layer bidirectional long short-term memory network connected in series are connected in sequence. The output of the two-layer bidirectional long short-term memory network connected in series is processed by a fully connected layer and a Softmax activation function and then connected to the connection time series classification network. The input of the connection time series classification network is the classification probability matrix processed by the fully connected layer and the Softmax activation function and the label l of the data set.

3. The continuous sign language recognition method based on the adaptive multi-scale attention time series network according to claim 1, characterized in that, The number of channels of the input feature dimension of the multi-scale time convolution module is 1024; the one-dimensional convolutions with different convolution kernel sizes are one-dimensional convolutions with convolution kernel sizes of 3, 5, 7, 9, 11 and padding sizes of 1, 2, 3, 4, 5; the kernel of the two-dimensional average pooling is [5, 1], and the kernel of the one-dimensional max pooling is 2.

4. A continuous sign language recognition method based on an adaptive multi-scale attention time series network, characterized in that, The specific steps of this method are as follows: First step, input the image sequence V and perform preprocessing: Input image sequence V=(v1,...,v i ,...,v T ), where T is the number of frames in the input image sequence V, which represent the first frame, ..., the i-th frame, ..., the T-th frame of the original sign language image sequence respectively. The size of the input video is 210×260, which is converted to 256×256, and then an image sequence W is obtained through preprocessing. The preprocessing process includes random cropping, random horizontal flipping, random scaling, and normalization to obtain the preprocessed sign language image sequence Second step, extract spatial features from the preprocessed image sequence W: Input the image sequence W obtained by preprocessing in the first step into the Resnet18 residual network for spatial feature extraction to obtain spatial features The dimension of F is (1, T, 1024), where the second dimension is the time dimension and the third dimension is the channel dimension; Third step, extract multi-scale spatio-temporal features of the spatial feature F: Input the spatial feature F obtained in the second step into the adaptive multi-scale attention time series network. The adaptive multi-scale attention time series network is composed of a multi-scale time convolution module, a channel attention module, and an adaptive multi-scale attention module. The specific operation of extracting the multi-scale spatio-temporal features of the spatial feature F is as follows: Input the spatial feature F obtained in the second step into the multi-scale temporal convolution module. The multi-scale temporal convolution module includes multiple parallel one-dimensional convolutions with different kernel sizes, stacking operations, and reshape functions. The specific structure is Conv1D_3, Conv1D_5, Conv1D_7, Conv1D_9, Conv1D_11, which represent one-dimensional convolutions with kernel sizes of 3, 5, 7, 9, 11 and padding sizes of 1, 2, 3, 4, 5 respectively. The spatial feature F passes through one-dimensional convolutions of five different scales to obtain spatio-temporal features I3, I5, I7, I9, I 11 with dimensions of (1, 1024, T) for all of them; combine the spatio-temporal features with different receptive fields obtained and then use the reshape function to get the combined spatio-temporal feature I′ with a feature dimension of (T, 1024, 5, 1); Input the obtained combined spatio-temporal feature I′ into the channel attention module for channel attention enhancement. The channel attention module includes two parallel average pooling and max pooling. The outputs of the average pooling and max pooling are respectively connected to a two-dimensional convolution with a convolution kernel size of [1, 1], a Relu activation function, and a two-dimensional convolution with a convolution kernel size of [1, 1]; then, after element-wise addition and Sigmoid activation function processing, the channel attention coefficient is obtained, and then, after performing a Hadamard product with the input of the channel attention module and passing through the reshape function, the combined spatio-temporal feature I″ with enhanced channel attention is obtained. The specific process is as follows: First, two-channel information features G are screened out through max pooling and average pooling respectively max and G avg , their dimensions are (T, 1024, 1, 1), and are expressed as shown in the following formulas (9) and (10). G max = MaxPool(I′) (9) G avg = AvgPool(I′) (10) Among them, MaxPool(·) represents the max pooling function, and AvgPool(·) represents the average pooling function. Two-channel information features G max and G avg are respectively passed through two-dimensional convolutions with a convolution kernel size of [1, 1] to reduce the channel dimension, then activated by the Relu activation function respectively, and then passed through two-dimensional convolutions with a convolution kernel size of [1, 1] to increase the channel dimension, obtaining G′ max and G′ avg , expressed as shown in the following formulas (11) and (12). G′ max = Conv2D 1×1 (Relu(Conv2D 1×1 (G max ))) (11) G′ avg = Conv2D 1×1 (Relu(Conv2D 1×1 (G avg ))) (12) Among them, Conv2D 1×1 (·) represents a two-dimensional convolution with a convolution kernel size of [1, 1], and Relu(·) represents the Relu activation function. Finally, add the corresponding elements of G′ obtained from formulas (11) and (12) max and G′ avg to generate channel attention coefficients through the Sigmoid activation function. After performing the Hadamard product between the channel attention coefficients and the combined spatio-temporal feature I′, then convert the dimension through the reshape function to obtain the combined spatio-temporal feature I″ with enhanced channel attention, whose feature dimension is (1, 1024, 5, T), expressed as shown in the following formula (13). I″ = reshape(I′⊙(Sigmoid(G′ max +G′ avg ))) (13) In formula (13), + represents the element-wise addition of two matrices, ⊙ represents the Hadamard product, Sigmoid(·) represents the Sigmoid activation function, and reshape(·) represents the function of converting the dimension of the feature vector. The combined spatio-temporal features I″ with enhanced channel attention obtained in formula (13) are input into the adaptive multi-scale attention module. The adaptive multi-scale attention module includes two parallel average pooling and max pooling. The outputs of the two are processed through a concatenation operation in the channel dimension, a two-dimensional convolution with a convolution kernel size of [5, 3] and a padding size of [2, 1], and a Sigmoid activation function to output the adaptive multi-scale attention coefficient Att mul , the adaptive multi-scale attention coefficient Att mul is subjected to a Hadamard product with the input of the adaptive multi-scale attention module, and then the adaptive multi-scale attention-enhanced combined spatio-temporal features I″′ are obtained through the reshape function. Finally, the multi-scale spatio-temporal features L are obtained through two-dimensional average pooling and one-dimensional max pooling in sequence; The specific process is as follows: First, two multi-scale information features R are respectively screened out through max pooling and average pooling in the channel dimension max and R avg , their feature dimensions are both (1, 1, 5, T), expressed as formulas (14) and (15). R max = MaxPool_c(I″) (14) R avg = AvgPool_c(I″) (15) In formulas (14) and (15), MaxPool_c(·) represents the max pooling function for the channel dimension, that is, the second dimension, and AvgPool_c(·) represents the average pooling function for the channel dimension, that is, the second dimension. Next, the two multi-scale information features R max and R avg are concatenated according to the channel dimension, and then convolution operation is performed through a two-dimensional convolution with a convolution kernel size of [5, 3] and a padding size of [2, 1]. Then, the Sigmoid activation function is used to obtain the adaptive multi-scale attention coefficient Att mul , whose feature dimension is (1, 1024, 5, T), expressed as formula (16). Att mul = Sigmoid(Conv2D 5×3 (contact(R max , R avg ))) (16) In formula (16), contact(·) represents a function that concatenates feature vectors along the channel dimension, and Conv2D 5×3 (·) represents a function of two-dimensional convolution with a convolution kernel size of [5, 3] and a padding size of [2, 1], and Sigmoid(·) represents the Sigmoid activation function. Finally, the adaptive multi-scale attention coefficient Att obtained in formula (16) mul and the combined spatio-temporal feature I″ enhanced by channel attention are subjected to Hadamard product, and then the combined spatio-temporal feature I′″ enhanced by adaptive multi-scale attention is obtained through the reshape function. Its feature dimension is (T, 1024, 5, 1), which is expressed as shown in the following formula (17). I″′ = reshape(I″ ⊙ Att mul ) (17) In formula (16), ⊙ represents the Hadamard product, and reshape(·) represents the function of converting the dimension of the feature vector. The combined spatio-temporal features I′″ enhanced by the adaptive multi-scale attention obtained in formula (17) are subjected to two-dimensional average pooling and one-dimensional max pooling to obtain multi-scale spatio-temporal features The dimension of its features is (T′, 1024), where T′ = T / 2, and its expression is shown in the following formula (18): AvgPool in formula (18) 2D (·) represents a two-dimensional average pooling function, and MaxPool 1D represents a one-dimensional maximum pooling function; Fourth step, extract the spatio-temporal features of the long-term time connection of the multi-scale spatio-temporal feature L: Input the multi-scale spatio-temporal feature L obtained in the above third step into a two-layer tandem bidirectional long short-term memory network to extract the spatio-temporal feature of its long-term temporal connection Fifth step, obtain the classification probability matrix according to the spatio-temporal features E of the long-term time connection: After inputting the spatio-temporal features E of the long-term time connection obtained in the fourth step into a fully connected layer, it is then activated through the Softmax activation function to obtain a classification probability matrix Its feature dimension is (T′, 1296). Each vector in Y is a probability value, and the sum of the probability values in the vector direction is 1. Step 6: Construct the connection timing classification loss function Train the neural network: Input the classification probability matrix obtained in the fifth step and the label l of the data set into the Connectionist Temporal Classification (CTC) network. The Connectionist Temporal Classification network introduces a blank label <blank>As an auxiliary marker, the predicted probability is obtained by summing the conditional probabilities of all possible paths, and the input is a classification probability matrix After that, the predicted probability P(l|Y) is obtained, and its expression is shown in the following formula (21).< / blank> In formula (21), ∏ = (π1, …, π i , … π s ) represents the set of all possible paths, π i represents the i-th possible path, s represents the total number of possible paths, B is a many-to-one mapping used to remove empty and duplicate labels in path ∏, B -1 is the inverse operation of B. Loss function is given by Equation (22) In formula (22), log(·) represents the logarithmic function with the natural number e as the base; In the seventh step, use the trained neural network to predict the sign language sequence and output the label H of the sign language sequence; Through the neural network trained in the sixth step, use beam search decoding to decode the classification probability matrix Y obtained in the fifth step to obtain the label H of the sign language sequence, which is expressed as formula (23), H = Beam(Y) (23) In formula (23), Beam(·) represents the beam search decoding function.

Images