A dynamic sign language recognition method based on spatiotemporal dual-flow feature fusion

Abstract

The application discloses a dynamic sign language recognition method based on space-time double-flow feature fusion. The specific steps are as follows: 1) a plurality of sign language videos are obtained from a sign language video data set, and all the sign language videos are classified by sign language, then each sign language video is preprocessed to obtain an image sequence, and all the preprocessed image sequences are divided into a training set and a test set; 2) a neural network model for dynamic sign language recognition is constructed, the neural network model comprises five time convolution modules, six space convolution modules, a pooling module, a multi-dimensional integration module and two average pooling layers; 3) the training set is input into the neural network model constructed in step 2) for training; 4) a sign language video to be recognized is preprocessed and then input into the trained neural network model for sign language recognition, and a classification recognition result is output. The application is designed according to the dynamic gesture recognition characteristics, and the accuracy of dynamic sign language recognition can be effectively improved.

Description

TECHNICAL FIELD

[0001] The application relates to an extraction method of isolated sign language video, in particular to a dynamic sign language recognition method based on spatio-temporal double-flow feature fusion. BACKGROUND

[0002] In recent years, with the development of deep learning, gesture recognition based on computer vision is becoming a hot trend. At present, the mainstream method for extracting sign language video is to rely on skeleton sequences, but the calculation of skeleton sequence information is completely independent of gesture recognition, which is suboptimal. And different skeleton extraction algorithms generate different skeleton sequence information for the same RGB image, which will cause differences in the final recognition result. More importantly, the pre-extraction of the skeleton increases the data preprocessing overhead, which causes difficulties for real-time inference of the network. In addition, the method based on depth image also plays a key role in gesture recognition, but the extraction of the depth image needs a special camera, and a general camera cannot achieve this, so it is difficult to deploy to most mobile devices. In recent years, some scholars have tried to use recurrent neural networks combined with convolutional neural networks or use traditional neural network methods to recognize sign language, but the problems are that the recognition accuracy is not high, and the performance in real life is not reliable. SUMMARY

[0003] In view of the limitations of the existing method in real-time inference, the application provides a dynamic sign language recognition method based on spatio-temporal double-flow feature fusion, which can effectively solve the difficulties of the existing method in deployment and inference, and is a new efficient and lightweight network model.

[0004] The technical scheme adopted by the application is as follows:

[0005] 1) Obtain a plurality of sign language videos from a sign language video data set, divide all the sign language videos into a training set and a test set, and then obtain an image sequence by preprocessing each sign language video;

[0006] 2) Construct a neural network model for dynamic sign language recognition, wherein the neural network model comprises five time convolution modules, six space convolution modules, one pooling module, one multi-dimensional integration module and two average pooling layers;

[0007] 3) input the training set into the neural network model constructed in step 2) for training

[0008] 4) preprocessing the sign language video to be recognized, and then inputting the trained neural network model for sign language recognition and outputting the classification recognition result.

[0009] The preprocessing process of steps 1) and 4) is as follows:

[0010] Frame the hand sign video and compress the image height and width, select 16 frames from the image sequence obtained after frame extraction in the strategy of sparse sampling, and stack them in time sequence, then randomly crop, separate the RGB three channels of the cropped image sequence, and normalize;

[0011] For the same sequence of images, the same cropping standard is applied: the center 256x256 of the image is taken as the cropping range, and an image with a size of 0.85-1 times the cropping range and an aspect ratio of 0.85-1.2 times the cropping range is randomly cut off, and the cut-off image is scaled to 224x224.

[0012] The neural network model structure of the step 2) is specifically:

[0013] The five time convolution modules are connected in turn, a pooling module is connected between the first time convolution module and the second time convolution module, and the output of the fifth time convolution module is obtained after the average pooling layer to obtain the fifth order time feature vector;

[0014] The input of the first spatial convolution module is connected to the output of the first time convolution module, the input of the second spatial convolution module is connected to the output of the pooling module, the input of the third spatial convolution module is connected to the output of the second time convolution module, and the input of the fourth spatial convolution module is connected to the output of the third time convolution module; the outputs of the first spatial convolution module, the second spatial convolution module, the third spatial convolution module and the fourth spatial convolution module are spliced in the channel dimension and input into the multi-dimensional integration module to obtain the integrated spatial feature vector; the integrated spatial feature vector is sequentially input into the fifth spatial convolution module, the sixth spatial convolution module and the average pooling layer to obtain the sixth order spatial feature vector;

[0015] The sixth order spatial feature vector and the fifth order time feature vector are spliced in the channel dimension and input into the softmax classifier, and the classification result of the softmax classifier is the final output of the model.

[0016] The first time convolution module is composed of a three-dimensional convolution layer, a normalization layer and a ReLU activation function connected in turn, wherein the convolution kernel size of the three-dimensional convolution layer is 1x7x7, the step is (1, 2, 2), and the padding is (0, 3, 3);

[0017] The second time convolution module, the third time convolution module, the fourth time convolution module and the fifth time convolution module are respectively composed of a three-dimensional convolution module with a depth of 3, a three-dimensional down-sampling convolution module with a depth of 4, a three-dimensional down-sampling convolution module with a depth of 6, and a three-dimensional down-sampling convolution module with a depth of 3;

[0018] The first spatial convolution module is sequentially connected by a three-dimensional convolution layer, a normalization layer and a ReLU activation function, wherein the three-dimensional convolution layer has a convolution kernel size of 3x7x7, a step of (2, 4, 4) and a padding of (1, 2, 2);

[0019] The second spatial convolution module, the third spatial convolution module, the fourth spatial convolution module, the fifth spatial convolution module and the sixth spatial convolution module are respectively composed of a three-dimensional down-sampling convolution module with a depth of 3, a three-dimensional down-sampling convolution module with a depth of 4, a three-dimensional convolution module with a depth of 6, a three-dimensional down-sampling convolution module with a depth of 2 and a three-dimensional down-sampling convolution module with a depth of 2.

[0020] The pooling module is a three-dimensional maximum pooling layer with a convolution kernel size of 1x3x3, a step of (1, 2, 2) and a padding of (0, 1, 1).

[0021] The three-dimensional convolution module with a depth of N is sequentially connected by N same bottleneck blocks; the three-dimensional down-sampling convolution module with a depth of N is sequentially connected by a down-sampling bottleneck block and N-1 same bottleneck blocks; N represents an Arabic numeral.

[0022] The bottleneck block is sequentially connected by three convolution blocks, and the initial input of the bottleneck block and the output of the last convolution block are added in the height and width dimensions to serve as the final output of the bottleneck block; each convolution block is sequentially connected by a three-dimensional convolution layer, a normalization layer and a ReLU activation function; the three-dimensional convolution layer in the first and third convolution blocks has a convolution kernel size of 1x1x1, a step of (1, 1, 1); the three-dimensional convolution layer in the second convolution block has a convolution kernel size of 1x3x3, a step of (1, 1, 1) and a padding of (0, 1, 1), and the number of groups is the number of input channels.

[0023] The down-sampling bottleneck block is sequentially connected by three convolution blocks and a parallel three-dimensional convolution layer, and the output of the parallel three-dimensional convolution layer and the output of the third convolution block are added to serve as the final output of the down-sampling bottleneck block; each convolution block is sequentially connected by a three-dimensional convolution layer, a normalization layer and a ReLU activation function; the three-dimensional convolution layer in the first and third convolution blocks has a convolution kernel size of 1x1x1, a step of (1, 1, 1); the three-dimensional convolution layer in the second convolution block has a convolution kernel size of 1x3x3, a step of (1, 2, 2) and a padding of (0, 1, 1), and the number of groups is the number of input channels; the parallel three-dimensional convolution layer has a convolution kernel size of 1x1x1 and a step of (1, 2, 2).

[0024] The multi-dimensional integration module comprises three channels, channel A, channel B and channel C, the outputs of channel A and channel B are connected to a first matrix multiplication layer, the output of the first matrix multiplication layer and channel C are connected to a second matrix multiplication layer, and the second matrix multiplication layer is sequentially connected with a deformation layer; each channel is sequentially connected with a three-dimensional convolution layer with a convolution kernel size of 1*1 and a deformation layer.

[0025] The structure of the multi-dimensional integration module is specifically as follows:

[0026] The input of the multi-dimensional integration module is represented by letter x, and the channel, time, height and width dimensions are represented by formula C*T*W*H;

[0027] Firstly, the vector x is input into the three-dimensional convolution layer A to obtain a vector with dimensions of C*T*W*d, and then the dimensions of the vector are deformed into C*T*H*W, which is represented by letter a; Secondly, the vector x is input into the three-dimensional convolution layer B to obtain a vector with dimensions of C*T*W*d, and then the dimensions of the vector are deformed into C*T*H*W, which is represented by letter b;

[0028] Thirdly, the vector x is input into the 1*1 three-dimensional convolution layer C to obtain a vector with dimensions of C*T*W*d, and then the dimensions of the vector are deformed into C*T*H*W, which is represented by letter d; Then, the vectors a and b are multiplied in matrix to obtain a vector with dimensions of C*T*H*W, which is represented by letter c; and the vectors c and d are multiplied in matrix to obtain a vector with dimensions of C*T*H*W;

[0029] Finally, the vector with dimensions of C*T*H*W is deformed to convert into an integrated spatial feature vector with dimensions of C*T*W*H.

[0030] The beneficial effects of the present application are as follows:

[0031] The method of the present application is designed according to the characteristics of dynamic gesture recognition, and the overall design adopts a structure combining spatial flow and motion flow to extract the spatial and motion features of gestures respectively. The time and space information of gestures can be modeled simultaneously, and the necessary expression features can be extracted. The fusion of space-time features can effectively improve the shortcomings of traditional convolution in dynamic gesture extraction.

[0032] The present application adopts a feature pyramid structure input mode for the spatial flow, and the feature map superimposes receptive fields of different ranges, which is sensitive to feature information of different scales, and can effectively extract the spatial information of gestures.

[0033] The method of the present application is designed according to the characteristics of dynamic gesture recognition, and the overall design adopts a structure combining spatial flow and motion flow to extract the spatial and motion features of gestures respectively. The time and space information of gestures can be modeled simultaneously, and the necessary expression features can be extracted. The fusion of space-time features can effectively improve the shortcomings of traditional convolution in dynamic gesture extraction.

[0034] The present application adopts a feature pyramid structure input mode for the spatial flow, and the feature map superimposes receptive fields of different ranges, which is sensitive to feature information of different scales, and can effectively extract the spatial information of gestures.

[0035] ​​The application only needs RGB video as input, and pre-extraction of skeleton sequence information is not needed.

[0036] The application adopts grouped convolution, the parameters of the model are reduced with the increase of the filter grouping number, the parameter quantity and the calculation quantity can be effectively reduced, and the efficiency of the model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The method of the application is implemented;

[0038] Figure 2 The sparse image sequence obtained by sparse sampling in the application;

[0039] Figure 3 The image sequence obtained by random cropping in the application;

[0040] Figure 4 The schematic diagram of the three-dimensional convolution module with a depth of N in the application;

[0041] Figure 5 The schematic diagram of the bottleneck block in the application;

[0042] Figure 6 The schematic diagram of the three-dimensional down-sampling convolution module with a depth of N in the application;

[0043] Figure 7 The schematic diagram of the down-sampling bottleneck block in the application;

[0044] Figure 8 The schematic diagram of the multi-dimensional integration module in the application. DETAILED DESCRIPTION

[0045] The application will be further described below in combination with the drawings and examples.

[0046] The specific implementation of the application is as follows:

[0047] The data set of the application adopts the Indian Lexicon Sign Language Dataset (INCLUDE). The Indian Lexicon Sign Language Dataset contains sign language videos under different angles, different illuminations, different testers and different backgrounds, and contains a total of 263 sign language words from 15 different word categories. The frame rate of the sign language video is 25 frames per second, the duration is 1-5 seconds, and the frame height and width are 1920 and 1088 respectively.

[0048] As shown in Figure 1 The implementation process includes the following steps

[0049] 1) Frame extraction and compression of the original video image height and width, obtain a continuous sequence of sign language images with a height of 256 and a width of 455, select 16 frames from the image sequence obtained after frame extraction in a sparse sampling strategy, and superimpose them in chronological order to obtain a group of sparse image sequences, which can be represented as 816x256x455. As shown in Figure 2

[0050] In the training phase, a small range of random cropping is performed on the image sequence to crop the image with a size of 0.85-1 times the range and a height-width ratio of 0.85-1.2 times the range, and scale it to 224x224. The cropped result is shown in Figure 3 For images in the same sequence, the same cropping standard needs to be applied.

[0051] Random cropping can expand the size of the data set, so that the same gesture can appear in different positions, thereby reducing the dependence of the model on the position of the gesture and optimizing the performance of the model on the data set.

[0052] Separate the cropped image sequence into RGB three channels and perform normalization processing to map each pixel point from an integer between 0 and 255 to a floating point number between 0 and 1.

[0053] 2) The cropped image sequence is input into the first temporal convolution module to obtain the original temporal feature vector.

[0054] In step 2):

[0055] The first temporal convolution module is composed of a three-dimensional convolution layer with a kernel size of 1x7x7, a stride of (1, 2, 2), and a padding of (0, 3, 3), a normalization layer, and a ReLU activation function connected in sequence.

[0056] The input feature vector dimension can be represented as 16x224 2 , representing that the input image sequence is a group of images with a height and width of 224x224, each containing 16 video frames.

[0057] The obtained original temporal feature vector dimension can be represented as 16x112 2 .

[0058] 3) The original temporal feature vector is input into the pooling module and the first spatial convolution module to obtain the first-order temporal feature vector and the first-order spatial feature vector.

[0059] In step 3):

[0060] ​The pooling module is composed of a three-dimensional max-pooling layer with a convolution kernel size of 1x3x3, a step of (1, 2, 2), and padding of (0, 1, 1). The dimension of the obtained first-order time feature vector can be represented as 16x56 2 .

[0061] The first spatial convolution module is composed of a three-dimensional convolution layer with a convolution kernel size of 3x7x7, a step of (2, 4, 4), and padding of (1, 2, 2), a normalization layer, and a ReLU activation function connected in sequence. The dimension of the obtained first-order spatial feature vector can be represented as 8x28 2 .

[0062] 4) The first-order time feature vector is input into the second time convolution module and the second spatial convolution module, respectively, to obtain the second-order time feature vector and the second-order spatial feature vector.

[0063] In step 4):

[0064] The second time convolution module is composed of a three-dimensional convolution module with a depth of 3, and the second spatial convolution module is composed of a three-dimensional down-sampling convolution module with a depth of 3.

[0065] The three-dimensional convolution module with a depth of N is composed of N identical bottleneck blocks connected in sequence, as shown in Figure 4 The three-dimensional convolution module receives an initial feature vector with a dimension of TxH 2 , and outputs a feature vector with a dimension of TxH 2 after being processed by the N identical bottleneck blocks in sequence.

[0066] The bottleneck block is mainly composed of a three-dimensional convolution layer with a convolution kernel size of 1x1x1, a step of (1, 1, 1), a normalization layer, and a ReLU activation function; a three-dimensional convolution layer with a convolution kernel size of 1x3x3, a step of (1, 1, 1), padding of (0, 1, 1), and a group number equal to the number of input channels, a normalization layer, and a ReLU activation function; a three-dimensional convolution layer with a convolution kernel size of 1x1x1, a step of (1, 1, 1), a normalization layer, and a ReLU activation function connected in sequence, as shown in Figure 5 .

[0067] Each layer is connected in sequence according to the above order, and a residual connection is introduced. The processing mode is that each layer connected in sequence processes the initial input vector in sequence, and the output of the last ReLU function is added to the initial input vector in the height and width dimensions to obtain the final output.

[0068] The three-dimensional down-sampling convolution module with a depth of N is composed of a down-sampling bottleneck block and N-1 identical bottleneck blocks connected in sequence, as shown in Figure 6The three-dimensional down-sampling convolution module receives an initial feature vector with a dimension of T x H 2 , outputs a feature vector with a dimension of T x (H / 2) 2 after processing by a down-sampling bottleneck block, and then goes through N-1 same bottleneck blocks, the input and output dimensions of which are all T x (H / 2) 2 .

[0069] The down-sampling bottleneck block mainly comprises a three-dimensional convolution layer with a kernel size of 1 x 1 x 1 and a step of (1, 1, 1), a normalization layer, and a ReLU activation function; a three-dimensional convolution layer with a kernel size of 1 x 3 x 3, a step of (1, 2, 2), padding of (0, 1, 1), and a group number equal to the input channel number, a normalization layer, and a ReLU activation function; and a three-dimensional convolution layer with a kernel size of 1 x 1 x 1 and a step of (1, 1, 1), a normalization layer, and a ReLU activation function connected in sequence, and a three-dimensional convolution layer with a kernel size of 1 x 1 x 1 and a step of (1, 2, 2) connected in parallel, as shown in Figure 7 .

[0070] The definition of the bottleneck block follows the method in step 4).

[0071] 5) The second-order time feature vector is respectively input into a third time convolution module and a third space convolution module to obtain a third-order time feature vector and a third-order space feature vector.

[0072] In step 5):

[0073] The third time convolution module and the third space convolution module are both composed of a three-dimensional down-sampling convolution module with a depth of 4. The definition of the three-dimensional down-sampling convolution module follows the method in step 4).

[0074] 6) The third-order time feature vector is respectively input into a fourth time convolution module and a fourth space convolution module to obtain a fourth-order time feature vector and a fourth-order space feature vector.

[0075] In step 6):

[0076] The fourth time convolution module and the fourth space convolution module are respectively composed of a three-dimensional down-sampling convolution module with a depth of 6 and a three-dimensional convolution module. The definitions of the three-dimensional down-sampling convolution module and the three-dimensional convolution module follow the method in step 4).

[0077] 7) The fourth-order time feature vector is input into a fifth time convolution module and then input into an average pooling layer to obtain a fifth-order time feature vector.

[0078] In step 7):

[0079] The fifth time convolution module is composed of a three-dimensional down-sampling convolution module with a depth of 3, and the definition of the three-dimensional down-sampling convolution module follows the method in step 4).

[0080] 8) The first-order spatial feature vector, the second-order spatial feature vector, the third-order spatial feature vector and the fourth-order spatial feature vector are spliced in the channel dimension as a fused spatial feature vector input into the multi-dimensional integration module to obtain an integrated spatial feature vector.

[0081] In step 8),

[0082] The multi-dimensional integration module is shown in Figure 8 and mainly composed of three three-dimensional convolution layers, four dimension transformation operations and two matrix multiplication operations.

[0083] The fused spatial feature vector is denoted by letter x, and its channel, time, height and width dimensions are denoted by formula C×T×W×H.

[0084] First, the vector x is input into a 1×1 three-dimensional convolution layer A to obtain a vector with a dimension of Then, the dimension of the vector is transformed into denoted by letter a.

[0085] Second, the vector x is input into a 1×1 three-dimensional convolution layer B to obtain a vector with a dimension of Then, the dimension of the vector is transformed into denoted by letter b.

[0086] The vectors a and b are multiplied by a matrix to obtain a vector with a dimension of CT×CT, denoted by letter c.

[0087] The vector x is input into a 1×1 three-dimensional convolution layer C to obtain a vector with a dimension of C×T×W×H. Then, the dimension of the vector is transformed into CT×HW, denoted by letter d.

[0088] The vectors c and d are multiplied by a matrix to obtain a vector with a dimension of CT×HW, and finally, a dimension transformation operation is performed to convert the dimension of the vector into an integrated spatial feature vector with a dimension of C×T×W×H.

[0089] 9) The integrated spatial feature vector is input into the fifth spatial convolution module to obtain a fifth-order spatial feature vector.

[0090] In step 9),

[0091] The fifth spatial convolution module is composed of a three-dimensional down-sampling convolution module with a depth of 2, and the definition of the three-dimensional down-sampling module follows the method in step 4).

[0092] 10) The fifth-order spatial feature vector is input into the sixth spatial convolution module and then input into an average pooling layer to obtain a sixth-order spatial feature vector.

[0093] In step 10) :

[0094] The sixth spatial convolution module is composed of a three-dimensional down-sampling convolution module with a depth of 2, and the definition of the three-dimensional down-sampling module follows the method in step 4).

[0095] 11) The fifth-order time feature vector and the sixth-order spatial feature vector are spliced in the channel dimension to obtain a seventh-order feature vector, which is input into a subsequent softmax classifier to obtain a final classification result.

[0096] As shown in Table 1, the present application can be better than the current most advanced algorithm on two different divisions of the INCLUDE dataset, indicating that the present application has excellent performance in dynamic gesture recognition.

[0097] Table 1 Performance of the present application compared with the optimal algorithm on the INCLUDE dataset

[0098] Prior best method Method of the invention INCLUDE 50 94.5％ 96.4％ INCLUDE 85.6％ 90.4％

Claims

1. A dynamic sign language recognition method based on spatiotemporal dual-flow feature fusion, characterized in that: The method comprises the following steps: 1) obtaining sign language videos from a sign language video dataset, dividing all the sign language videos into a training set and a test set, and then preprocessing each sign language video to obtain an image sequence; 2) constructing a neural network model for dynamic sign language recognition and training; inputting the preprocessed sign language video to be recognized into the trained model to output a classification result; wherein the structure of the neural network model is as follows: a) a time branch: five time convolution modules are connected in sequence, and a pooling module is connected between the first and second time convolution modules; the fifth time convolution module outputs a fifth-order time feature vector obtained through average pooling; the first time convolution module comprises, in sequence, a three-dimensional convolution layer, a normalization layer and ReLU, and the three-dimensional convolution layer has a convolution kernel size of 1x7x7, a step of (1, 2, 2), and padding of (0, 3, 3); the second, third, fourth and fifth time convolution modules are respectively composed of a three-dimensional convolution module with a depth of 3, a three-dimensional down-sampling convolution module with a depth of 4, a three-dimensional down-sampling convolution module with a depth of 6, and a three-dimensional down-sampling convolution module with a depth of 3; b) a spatial branch: the first spatial convolution module inputs the output of the first time convolution module, the second spatial convolution module inputs the output of the pooling module, and the third and fourth spatial convolution modules input the outputs of the second and third time convolution modules, respectively; the outputs of the first to fourth spatial convolution modules are concatenated in the channel dimension and then input into a multi-dimensional integration module to obtain an integrated spatial feature vector; the integrated spatial feature vector is sequentially input into a fifth and sixth spatial convolution module and average pooling to obtain a sixth-order spatial feature vector; the first spatial convolution module comprises, in sequence, a three-dimensional convolution layer, a normalization layer and ReLU, and the three-dimensional convolution layer has a convolution kernel size of 3x7x7, a step of (2, 4, 4), and padding of (1, 2, 2); the second to sixth spatial convolution modules are respectively composed of a three-dimensional down-sampling convolution module with a depth of 3, a three-dimensional down-sampling convolution module with a depth of 4, a three-dimensional convolution module with a depth of 6, a three-dimensional down-sampling convolution module with a depth of 2, and a three-dimensional down-sampling convolution module with a depth of 2; c) the pooling module is a three-dimensional max-pooling layer with a convolution kernel of 1x3x3, a step of (1, 2, 2), and padding of (0, 1, 1); d) the multi-dimensional integration module comprises three parallel channels A, B and C; each channel comprises, in sequence, a three-dimensional convolution layer with a size of 1x1 and a deformation layer; the outputs of channels A and B are connected to a first matrix multiplication layer, the output of the first matrix multiplication layer and the output of channel C are connected to a second matrix multiplication layer, and the second matrix multiplication layer is sequentially connected to a deformation layer to obtain the integrated spatial feature vector; the specific processing is as follows: the concatenated spatial feature vector is respectively processed through three 1x1x1 three-dimensional convolutions to obtain three groups of vectors, each of which is deformed into a two-dimensional matrix for matrix multiplication; the matrices of channels A and B are multiplied first, and then the obtained matrix and the matrix of channel C are multiplied, and then the result matrix is deformed into the integrated spatial feature vector. e) the sixth-order spatial feature vector is spliced with the fifth-order time feature vector in the channel dimension, and the spliced result is input into a softmax classifier to obtain a final output.

2. The dynamic sign language recognition method based on spatio-temporal dual-flow feature fusion according to claim 1, characterized in that: The preprocessing process is specifically: Frame extraction and image height-width compression are performed on the sign language video, 16 frames are selected from the image sequence obtained after frame extraction in a sparse sampling strategy, and the selected frames are stacked in time sequence and randomly cropped. For images in the same sequence, the same cropping standard is applied: the center 256x256 of the image is taken as the cropping range, an image with a size of 0.85-1 times the cropping range and an aspect ratio of 0.85-1.2 times the cropping range is randomly cropped, and the cropped image is scaled to 224x224.

3. The dynamic sign language recognition method based on spatio-temporal dual-flow feature fusion according to claim 1, wherein: The three-dimensional convolution module with a depth of N is composed of N same bottleneck blocks connected in sequence; The three-dimensional down-sampling convolution module with a depth of N is composed of one down-sampling bottleneck block and N-1 same bottleneck blocks connected in sequence; N represents an Arabic numeral.

4. The dynamic sign language recognition method based on spatio-temporal dual-flow feature fusion according to claim 1, wherein: The bottleneck block is composed of three convolution blocks connected in sequence, and the initial input of the bottleneck block and the output of the last convolution block are added in the height-width dimension to serve as the final output of the bottleneck block through the introduction of a residual connection; Each convolution block is composed of a three-dimensional convolution layer, a normalization layer, and a ReLU activation function connected in sequence; the three-dimensional convolution layer in the first and third convolution blocks has a kernel size of 1x1x1 and a step of (1, 1, 1), the three-dimensional convolution layer in the second convolution block has a kernel size of 1x3x3, a step of (1, 1, 1), a padding of (0, 1, 1), and a group number equal to the number of input channels.

5. The dynamic sign language recognition method based on spatio-temporal dual-flow feature fusion according to claim 1, wherein: The down-sampling bottleneck block is composed of three convolution blocks connected in sequence and a parallel three-dimensional convolution layer, and the output of the parallel three-dimensional convolution layer and the output of the third convolution block are added to serve as the final output of the down-sampling bottleneck block; Each convolution block is composed of a three-dimensional convolution layer, a normalization layer, and a ReLU activation function connected in sequence; the three-dimensional convolution layer in the first and third convolution blocks has a kernel size of 1x1x1 and a step of (1, 1, 1), the three-dimensional convolution layer in the second convolution block has a kernel size of 1x3x3, a step of (1, 2, 2), a padding of (0, 1, 1), and a group number equal to the number of input channels; and the parallel three-dimensional convolution layer has a kernel size of 1x1x1 and a step of (1, 2, 2).

Images