Continuous sign language recognition method based on multi-scale space and motion feature extraction

Abstract

The invention belongs to the technical field of sign language recognition, and particularly relates to a continuous sign language recognition method based on multi-scale space and motion feature extraction. On the basis of the Resnet18 network, a multi-scale spatial feature extraction module and a multi-scale motion feature extraction module which are parallel are embedded between two convolution blocks of each residual connection block of the Resnet18 network, and after final spatial features output by the multi-scale spatial feature extraction module and final motion features output by the multi-scale motion feature extraction module are added, the final motion features are extracted by the multi-scale motion feature extraction module. The feature is used as an input feature of a second convolution block of the residual connection block; sequentially performing dimension conversion, dimension remodeling, global average pooling, dimension remodeling and a full connection layer on an output feature of a fourth residual connection block of the Resnet18 network to obtain a feature O5; the feature O5 sequentially passes through a one-dimensional time convolution layer and two bidirectional long and short time memory network layers to obtain long-term time-related spatial-temporal features; the spatio-temporal features of long-term time connection sequentially pass through a full connection layer and a Softmax activation function to obtain a classification probability matrix; and decoding the classification probability matrix to obtain a sign language recognition result. According to the method, subtle local features, rapid changes of sign language actions, long-term trends and overall structures can be captured, and the accuracy and robustness of recognition are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and pattern recognition, and in particular is a continuous sign language recognition method based on multi-scale space and motion feature extraction. Background Art

[0002] Continuous sign language recognition technology refers to receiving a sign language video as input and converting the video content into corresponding sign language sentences. The basic building block of sign language is called "gloss", which combines expressions, gestures and movements to convey specific meanings. Traditional sign language recognition methods are highly dependent on manual features extracted from sign language videos or images, which usually require expert knowledge to design and optimize. In contrast, deep learning-based sign language recognition methods automatically learn and extract the features required for continuous sign language recognition through models, and can handle more complex and changeable sign language data.

[0003] The invention application with the publication number CN118537916A discloses a method and device for sign language recognition based on full-body features using a spatiotemporal cross-attention network. First, an RGB video is input into a full-body posture estimator to obtain the target position coordinates of the key points of the human body. These coordinates are used to generate an initial heat map of the skeleton. Then, the first original feature, the first interactive feature, the second original feature, and the second interactive feature are extracted from the RGB video and the initial heat map respectively. These features are then input into a spatiotemporal cross-attention module for optimization processing to generate the first optimized feature and the second optimized feature. Finally, the two optimized features are spliced ​​and input into a classifier to obtain the sign language recognition result. However, this method relies on full-body posture estimation and human key point information, and has limited ability to capture subtle local features (such as facial expressions) in sign language expressions. In the process of feature extraction and fusion, although the spatiotemporal cross-attention mechanism is introduced, it is still insufficient in processing multi-scale space and motion features, which may lead to the loss or under-utilization of some important features. Summary of the invention

[0004] In view of the deficiencies of the prior art, the technical problem to be solved by the present invention is: to provide a continuous sign language recognition method based on multi-scale spatial and motion feature extraction. Multi-scale spatial features are extracted through multi-scale dilated convolution, multi-scale motion features are obtained through frame-by-frame difference and cross-frame difference, and attention coefficients are given to multi-scale spatial and motion features respectively using an attention mechanism. Finally, through the fusion of spatial features and motion features, the problem of insufficient feature information caused by using a single spatial feature or motion feature is solved, and the robustness and recognition performance of the model are enhanced.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] A continuous sign language recognition method based on multi-scale space and motion feature extraction, characterized in that the method comprises the following steps:

[0007] Step 1: Obtain a sign language video, pre-process the sign language video, and obtain a pre-processed sign language image sequence;

[0008] Step 2: Construct a sign language recognition model; embed a parallel multi-scale spatial feature extraction module and a multi-scale motion feature extraction module between the two convolution blocks of each residual connection block of the Resnet18 network; add the final spatial features output by the multi-scale spatial feature extraction module and the final motion features output by the multi-scale motion feature extraction module, and use them as the input features of the second convolution block of the residual connection block; the output features of the fourth residual connection block of the Resnet18 network are successively subjected to dimension conversion, dimension reshaping, global average pooling, dimension reshaping and full connection layer to obtain feature O5; feature O5 is successively subjected to a one-dimensional time convolution layer and two bidirectional long short-term memory network layers to obtain the spatiotemporal features of long-term temporal connections; the spatiotemporal features of long-term temporal connections are successively subjected to the full connection layer and the Softmax activation function to obtain the classification probability matrix; the classification probability matrix is ​​decoded to obtain the sign language recognition result;

[0009] The input features of the multi-scale spatial feature extraction module are subjected to channel dimension reduction by three-dimensional convolution to obtain feature X'; feature X' is subjected to three different scales of dilated convolution respectively to obtain three features X'1, X'2 and X'3; features X'1, X'2 and X'3 are added element by element to obtain the added feature; the added feature is subjected to channel dimension increase by three-dimensional convolution, and the features after channel dimension increase are added element by element with the input features of the multi-scale spatial feature extraction module to obtain the preliminary spatial feature; the preliminary spatial feature is subjected to fast Fourier transform, and then the spatial attention coefficient is generated by the Sigmoid activation function, and the spatial attention coefficient is added to the preliminary spatial feature after the Hadamard product is made with the preliminary spatial feature, and then the final spatial feature is obtained by adding the preliminary spatial feature element by element;

[0010] Step 3: Train the sign language recognition model and use the trained sign language recognition model for continuous sign language recognition.

[0011] Furthermore, the input features of the multi-scale motion feature extraction module are successively subjected to global average pooling and one-dimensional convolution to obtain feature X'; feature X' is input into the fast motion feature extraction branch, and frame-by-frame difference calculation is performed according to formula (11) and all differential features are spliced ​​in the channel dimension to obtain feature X f ;

[0012] X f=Concat(X”(2)-X”(1),...,X”(t+1)-X”(t),...,X”(T)-X”(T-1)) (11)

[0013] Where Concat(.) represents the concatenation operation, X”(t) and X”(t+1) represent the features of feature X” on the tth and t+1th frames respectively, and T is the number of frames;

[0014] The feature X f Pad zeros in the time dimension to make it consistent with feature X”; then, use two one-dimensional convolutions to capture short-term time information and restore channels respectively, and increase the height and width dimensions to obtain fast motion features;

[0015] Input feature X' into the slow motion feature extraction branch, downsample feature X' along the frame number dimension to generate feature X''; perform frame-by-frame differential calculation on feature X'', and concatenate all differential features according to the channel dimension to obtain feature X s ; Set feature X s By upsampling, the time dimension is made consistent with the fast motion features. Then, two one-dimensional convolutions are performed to capture short-term time information and restore channels respectively, and the height and width dimensions are increased to obtain slow motion features.

[0016] The fast motion features and the slow motion features are added element by element to obtain the fast-slow fusion motion features. The fast-slow fusion motion features are activated by the Sigmoid function to generate the motion attention coefficient. The motion attention coefficient is Hadamard-producted with the input features of the multi-scale motion feature extraction module, and then added element by element to the input features to obtain the final motion features.

[0017] Furthermore, in the first step, preprocessing includes image resizing, random cropping, random horizontal flipping, random scaling, and normalization.

[0018] Furthermore, in the third step, the classification probability matrix is ​​input into the connection time series classification network, and the predicted probability is obtained as:

[0019]

[0020] In the formula, P(l|Y) represents the predicted probability, P(Π|Y) represents the conditional probability of the possible path, represents the set of all possible paths, π i represents the i-th possible path, N represents the total number of possible paths, B is a many-to-one mapping used to remove blank labels and duplicate labels in the set Π, B -1 is the inverse mapping of B; l is the label of the data set;

[0021] The training loss is calculated using the CTC loss function of formula (20):

[0022] L CTC = -lnP(l|Y) (20)

[0023] Where, L CTC is the CTC loss.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) The multi-scale spatial feature extraction module obtains multi-scale spatial features through multi-scale dilated convolution. By introducing fast Fourier transform, the features are converted from the spatial domain to the frequency domain and then back to the spatial domain. This not only broadens the dimension of feature extraction, but also enables the model to capture subtle local features that are difficult to obtain in spatial domain analysis, such as the dynamic frequency characteristics of gestures. This effectively solves the problem of low recognition accuracy caused by subtle differences in movements in sign language recognition, and significantly improves the accuracy and robustness of sign language recognition.

[0026] (2) Another major technical challenge of sign language recognition is how to efficiently and accurately capture the temporal dynamic features of sign language movements. Although the existing two-stream network has made some contributions in this regard, its high computational cost limits its practical application. To address this problem, the multi-scale motion feature extraction module of the present invention is divided into two branches: fast motion feature extraction and slow motion feature extraction. The fast motion feature extraction branch focuses on capturing rapid changes in sign language movements, such as rapid movement of fingers and rapid switching of gestures, while the slow motion feature extraction branch focuses on analyzing the long-term trend and overall structure of the movement. This design not only avoids the high computing resources required by the two-stream network, but also achieves a more comprehensive and detailed understanding of sign language movements through the fusion of fast and slow motion features, thereby significantly improving the performance of sign language recognition while maintaining computational efficiency.

[0027] (3) The method of the present invention can more comprehensively capture and utilize the feature information in the sign language expression, thereby improving the recognition performance of the model. The multi-scale spatial feature extraction module and the multi-scale motion feature extraction module are highly modular and plug-and-play, and can be easily integrated into the backbone network of other sign language recognition. The amount of differential calculation is much less than that of the commonly used optical flow method, which enables the method of the present invention to achieve higher computational efficiency while maintaining a high recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is the overall flow chart of the present invention;

[0029] Figure 2 is a schematic diagram of the structure of the multi-scale spatial feature extraction module of the present invention;

[0030] Figure 3 It is a structural schematic diagram of the multi-scale motion feature extraction module of the present invention. DETAILED DESCRIPTION

[0031] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to introduce the technical solutions of the present invention in detail and are not intended to limit the protection scope of the present application.

[0032] The present invention provides a continuous sign language recognition method based on multi-scale space and motion feature extraction (hereinafter referred to as method, see Figures 1 to 3 ), including the following steps:

[0033] Step 1: Obtain a sign language video, pre-process the sign language video, and obtain a pre-processed sign language image sequence;

[0034] The sign language video is denoted as V = (v1,…,v t ,…v T ), where T is the number of frames, v t Represents the original sign language image of the tth frame; the original sign language images of all frames are resized to 256×256 pixels, and then randomly cropped to 224×224, randomly flipped horizontally, randomly scaled and normalized to obtain the preprocessed sign language image sequence Its feature dimensions are (N, C, T, H, W), i t represents the preprocessed sign language image of the tth frame. The first dimension is the batch dimension, the second dimension is the channel dimension, the third dimension is the number of frames, the fourth dimension is the height dimension, and the fifth dimension is the width dimension. The preprocessing process is shown in formula (1):

[0035] i t =Normalize(RScale(RHFlip(RCrop(Resize(v t )))))(1)

[0036] In formula (1), Normalizate(.) represents the normalization operation, RScale(.) represents the random scaling operation, RHFlip(.) represents the random horizontal flipping operation, RCrop(.) represents the random cropping operation, and Resize(.) represents the image resizing operation.

[0037] Step 2: Build a sign language recognition model; based on the Resnet18 network, embed a parallel multi-scale spatial feature extraction module and a multi-scale motion feature extraction module between two convolution blocks of each residual connection block of the Resnet18 network;

[0038] The input feature X of the multi-scale spatial feature extraction module is subjected to channel dimensionality reduction through a three-dimensional convolution with a convolution kernel size of [1,1,1] and a padding size of [0,0,0], reducing the channel dimension to 1 / 16 of the original to reduce the amount of calculation, and obtaining the feature X', whose feature dimension is (N, 4, T, H / 4, W / 4); channel dimensionality reduction can be expressed as follows:

[0039] X' = ​​Conv3D 1×1×1 (X) (2)

[0040] In the formula, Conv3D 1×1×1 (.) represents a three-dimensional convolution with a kernel size of [1,1,1] and a padding size of [0,0,0];

[0041] The feature X' is subjected to three dilated convolutions of different scales to obtain three features X'1, X'2 and X'3, which can be expressed by the following formula:

[0042] X'1=Conv1(X')×α1 (3)

[0043] X'2=Conv2(X')×α2 (4)

[0044] X'3=Conv3(X')×α3 (5)

[0045] Where α1, α2, and α3 are learnable weight factors, which are initialized to 0 and updated through back-propagation during training to better capture the complex relationship between features; Conv1(.), Conv2(.), and Conv3(.) represent dilated convolutions with kernel sizes of [3,3,3] and dilation rates of [1,1,1], [1,2,2], and [1,3,3], respectively;

[0046] Add features X'1, X'2, and X'3 element by element to get the added feature X add ; Add feature X add The channel dimension is increased by three-dimensional convolution with a convolution kernel size of [1,1,1] and a padding size of [0,0,0] to restore the original channel dimension. Its feature dimension is (N,64,T,H / 4,W / 4), which can be expressed as:

[0047] X add =X'1+X'2+X'3 (6)

[0048] X' add =Conv3D 1×1×1 (X add ) (7)

[0049] Where X' add It is the added feature after channel dimension upgrading;

[0050] The feature X' after channel dimension upgrade add Add the input feature X of the multi-scale spatial feature extraction module element by element to obtain the preliminary spatial feature X spatial , expressed as:

[0051] X spatial =X' add +X (8)

[0052] The preliminary spatial feature X spatial After fast Fourier transform, the Sigmoid activation function is used to generate the spatial attention coefficient. The spatial attention coefficient is related to the preliminary spatial feature X spatial After making the Hadamard product, it is then combined with the preliminary spatial feature X spatial Add element by element to get the final spatial feature X' spatial , whose feature dimension is (N, 64, T, H / 4, W / 4); it can be expressed as:

[0053] X' spatial =Sigmoid(FFT(X spatial ))⊙X spatial +X spatial (9)

[0054] Where FFT(.) represents fast Fourier transform, Sigmoid(.) represents Sigmoid activation function, and ⊙ represents Hadamard product.

[0055] The input feature X of the multi-scale motion feature extraction module eliminates the spatial and temporal dimensions through the global average pooling operation, and then performs channel dimension reduction through a one-dimensional convolution with a convolution kernel size of 1 and a padding size of 0, reducing the channel dimension to 1 / 16 of the original, and obtaining feature X", whose feature dimension is (N, 4, T), which can be expressed as:

[0056] X' = ​​Conv1D k＝1 (AvgPool(X)) (10)

[0057] In the formula, Conv1D k＝1 (.) represents a one-dimensional convolution with a kernel size of 1 and a padding size of 0, and AvgPool(.) represents a global average pooling operation;

[0058] Input feature X" into the fast motion feature extraction branch, perform frame-by-frame difference calculation, and concatenate all differential features in the channel dimension to obtain feature X f , whose feature dimension is (N,8,T-1), expressed as:

[0059] X f=Concat(X”(2)-X”(1),...,X”(t+1)-X”(t),...,X”(T)-X”(T-1)) (11)

[0060] Where Concat(.) represents the concatenation operation, X”(t) and X”(t+1) represent the features of feature X” at the tth and t+1th frames respectively;

[0061] The feature X f Pad the time dimension with zeros to make it consistent with feature X'; then, capture short-term time information through one-dimensional convolution with a kernel size of 5 and a padding size of 2, and then perform channel dimension upscaling through one-dimensional convolution with a kernel size of 1 and a padding size of 0 to restore the channel dimension and increase the height and width dimensions to obtain the fast motion feature X' f , whose feature dimension is (N,64,T,1,1), which can be expressed as:

[0062] X' f =Conv1D k＝1 (Conv1D k＝5 (Concat(X f ,0))) (12)

[0063] In the formula, 0 represents the shape and feature X f Same all-zero matrix, Conv1D k＝1 (.) indicates a one-dimensional convolution with a kernel size of 1 and a padding size of 0, Conv1D k＝5 (.) represents a one-dimensional convolution with a kernel size of 5 and a padding size of 2.

[0064] The feature X' is input into the slow motion feature extraction branch, and the feature X' is downsampled along the frame number dimension, that is, frames are taken at a certain time step (the time step in this embodiment is 4 frames), thereby generating feature X'', whose feature dimension is The formula is:

[0065] X”'=Downsampling(X”) (13)

[0066] Where Downsampling(.) represents the downsampling operation along the frame number dimension;

[0067] Perform frame-by-frame differential calculation on feature X'', and concatenate all differential features according to the channel dimension to obtain feature X s , whose characteristic dimension is

[0068] The feature X sBy upsampling, its time dimension is consistent with the fast motion feature X' f The time dimension of is consistent, and then the short-term time information is captured by a one-dimensional convolution with a convolution kernel size of 5 and a padding size of 2. Then, the channel dimension is restored by a one-dimensional convolution with a kernel size of 1 and a padding size of 0, and the height and width dimensions are increased to obtain the slow motion feature X' s , whose feature dimension is (N,64,T,1,1), which can be expressed as:

[0069] X' s =Conv1D k＝1 (Conv1D k＝5 (Upsampling(X s ))) (14)

[0070] In the formula, Upsampling(.) represents the upsampling operation in the time dimension;

[0071] The fast motion feature X' f and slow motion feature X' s The two are added element by element to achieve the fusion of fast and slow motion features, and the fast and slow fusion motion feature X is obtained. motion , whose feature dimension is (N,64,T,1,1), which can be expressed as:

[0072] X motion =X' f +X' s (15)

[0073] Fusion of fast and slow motion features X motion The motion attention coefficient is generated by the Sigmoid activation function. After the motion attention coefficient is Hadamard-producted with the input feature X of the multi-scale motion feature extraction module, it is added element by element to obtain the final motion feature X' motion , whose feature dimension is (N, 64, T, H / 4, W / 4), which can be expressed as:

[0074] X' motion =Sigmoid(X motion )⊙X+X (16)

[0075] The final spatial feature X' spatial With the final motion feature X' motion After addition, they serve as the input features of the second convolutional block of the residual connection block.

[0076] The output feature of the residual connection block Conv2_x of the Resnet18 network is recorded as O1, and its feature dimension is (N, 64, T, H / 4, W / 4); the output feature of the residual connection block Conv3_x is recorded as O2, and its feature dimension is (N, 128, T, H / 8, W / 8); the output feature of the residual connection block Conv4_x is recorded as O3, and its feature dimension is (N, 256, T, H / 16, W / 16); the output feature of the residual connection block Conv5_x is recorded as O4, and its feature dimension is (N, 512, T, H / 32, W / 32);

[0077] The dimension of the output feature O4 of the residual connection block Conv5_x is converted to (N, T, 512, H / 32, W / 32), and then the dimension is reshaped to (N×T, 512, H / 32, W / 32). After that, global average pooling is performed, and its feature dimension is (N×T, 512, 1, 1); then, the dimension is reshaped again to (N×T, 512), and finally through the fully connected layer, the feature O5 is obtained, and its feature dimension is (N×T, C'), where C' is the number of categories, which is expressed by the formula:

[0078] O5=FC(AvgPool(View(Transpose(O4))))(17)

[0079] In the formula, FC(.) represents the fully connected layer, AvgPool(.) represents the global average pooling operation, View(.) represents the dimension reshaping operation, and Transpose(.) represents the dimension transformation operation;

[0080] Feature O5 uses a one-dimensional temporal convolutional layer to extract local visual information, and then passes through two bidirectional long short-term memory network layers in series to extract the spatiotemporal features of long-term temporal connections. Where T' is the time dimension of feature O5 after passing through the one-dimensional time convolution layer, which can be expressed as:

[0081] E=BiLSTM(TemporalConv(O5)) (18)

[0082] In the formula, BiLSTM(.) represents the serialized bidirectional long short-term memory network layer, and TemporalConv(.) represents the one-dimensional temporal convolution layer;

[0083] The spatiotemporal features of long-term temporal connections are sequentially passed through the fully connected layer and the Softmax activation function to obtain the classification probability matrix Its feature dimension is (T', 1296), where each vector is a probability value, and the sum of the probability values ​​in the vector direction is 1; the classification probability matrix is ​​decoded to obtain the sign language recognition result.

[0084] Step 3: Train the sign language recognition model and input the classification probability matrix Y into the Connectionist Temporal Classification (CTC) network. The Connectionist Temporal Classification network outputs the predicted probability, which can be expressed as:

[0085]

[0086] In the formula, P(l|Y) represents the predicted probability, P(Π|Y) represents the conditional probability of the possible path, represents the set of all possible paths, π i represents the i-th possible path, N represents the total number of possible paths, B is a many-to-one mapping used to remove blank labels and duplicate labels in the set Π, B -1 is the inverse mapping of B; l is the label of the data set;

[0087] The goal of the connected temporal classification network is to maximize the prediction probability, so the CTC loss function calculates the training loss. The CTC loss function is:

[0088] L CTC = -lnP(l|Y) (20)

[0089] The sign language image sequence to be recognized is input into the trained sign language recognition model to obtain a classification probability matrix; the classification probability matrix is ​​decoded by beam search decoding to obtain the label of the sign language image sequence, that is, the sign language recognition result.

[0090] The above-mentioned bidirectional long short-term memory network (Bi-directional Long-Short Term Memory, BiLSTM), connectionist temporal classification network (Connectionist Temporal Classification, CTC) and fast Fourier transform (Fast Fourier Transform, FFT) are all well-known technologies.

[0091] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A continuous sign language recognition method based on multi-scale space and motion feature extraction, characterized in that: The method comprises the following steps: Step 1: Obtain a sign language video, pre-process the sign language video, and obtain a pre-processed sign language image sequence; Step 2: Construct a sign language recognition model; embed a parallel multi-scale spatial feature extraction module and a multi-scale motion feature extraction module between the two convolution blocks of each residual connection block of the Resnet18 network; add the final spatial features output by the multi-scale spatial feature extraction module and the final motion features output by the multi-scale motion feature extraction module, and use them as the input features of the second convolution block of the residual connection block; the output features of the fourth residual connection block of the Resnet18 network are successively subjected to dimension conversion, dimension reshaping, global average pooling, dimension reshaping and full connection layer to obtain feature O5; feature O5 is successively subjected to a one-dimensional time convolution layer and two bidirectional long short-term memory network layers to obtain the spatiotemporal features of long-term temporal connections; the spatiotemporal features of long-term temporal connections are successively subjected to the full connection layer and the Softmax activation function to obtain the classification probability matrix; the classification probability matrix is ​​decoded to obtain the sign language recognition result; The input features of the multi-scale spatial feature extraction module are subjected to channel dimension reduction through three-dimensional convolution to obtain feature X'; feature X' is subjected to three different scales of dilated convolution respectively to obtain three features X1', X'2 and X3'; features X1', X'2 and X3' are added element by element to obtain added features; the added features are subjected to channel dimension increase through three-dimensional convolution, and the features after channel dimension increase are added element by element to the input features of the multi-scale spatial feature extraction module to obtain preliminary spatial features; the preliminary spatial features are subjected to fast Fourier transform, and then the spatial attention coefficient is generated through the Sigmoid activation function, and the spatial attention coefficient is added to the preliminary spatial features after the Hadamard product is made, and then the final spatial features are obtained by adding the preliminary spatial features element by element; Step 3: Train the sign language recognition model and use the trained sign language recognition model for continuous sign language recognition.

2. The continuous sign language recognition method based on multi-scale space and motion feature extraction according to claim 1 is characterized in that: The input features of the multi-scale motion feature extraction module are sequentially subjected to global average pooling and one-dimensional convolution to obtain feature X”; Input feature X' into the fast motion feature extraction branch, perform frame-by-frame differential calculation according to formula (11) and concatenate all differential features in the channel dimension to obtain feature X f ; X f =Concat(X”(2)-X”(1),...,X”(t+1)-X”(t),...,X”(T)-X”(T-1)) (11) Where Concat(.) represents the concatenation operation, X”(t) and X”(t+1) represent the features of feature X” on the tth and t+1th frames respectively, and T is the number of frames; The feature X f Pad zeros in the time dimension to make it consistent with feature X”; then, use two one-dimensional convolutions to capture short-term time information and restore channels respectively, and increase the height and width dimensions to obtain fast motion features; Input feature X' into the slow motion feature extraction branch, downsample feature X' along the frame number dimension to generate feature X''; Perform frame-by-frame differential calculation on feature X'' and concatenate all differential features according to the channel dimension to obtain feature X s ; Set feature X s By upsampling, the time dimension is made consistent with the fast motion features. Then, two one-dimensional convolutions are performed to capture short-term time information and restore channels respectively, and the height and width dimensions are increased to obtain slow motion features. The fast motion features and the slow motion features are added element by element to obtain the fast-slow fusion motion features. The fast-slow fusion motion features are activated by the Sigmoid function to generate the motion attention coefficient. The motion attention coefficient is Hadamard-producted with the input features of the multi-scale motion feature extraction module, and then added element by element to the input features to obtain the final motion features.

3. The continuous sign language recognition method based on multi-scale space and motion feature extraction according to claim 1 or 2, characterized in that: In the first step, preprocessing includes image resizing, random cropping, random horizontal flipping, random scaling, and normalization.

4. The continuous sign language recognition method based on multi-scale space and motion feature extraction according to claim 1, characterized in that: In the third step, the classification probability matrix is ​​input into the connection time series classification network, and the predicted probability is obtained as: In the formula, P(l|Y) represents the predicted probability, P(Π|Y) represents the conditional probability of the possible path, represents the set of all possible paths, π i represents the i-th possible path, N represents the total number of possible paths, B is a many-to-one mapping used to remove blank labels and duplicate labels in the set Π, B -1 is the inverse mapping of B; l is the label of the data set; The training loss is calculated using the CTC loss function of formula (20): L CTC =-lnP(l|Y) (20) Where, L CTC is the CTC loss.

Images