A 3D-ResNet-based sign language recognition method

Abstract

The application relates to the technical field of sign language recognition, and discloses a sign language recognition method based on 3D-ResNet, which comprises the following steps: input feature extraction is carried out on an image sequence through a 3D small convolution residual module and a maximum pooling layer; extracted input features are sequentially input through four pooling convolution residual blocks to extract high-level feature information; and the high-level feature information obtained in the last step is input into a softmax through an average pooling layer and a full connection layer for classification output. The application can more effectively extract low-level features, strengthen the weight of useful features, weaken the influence of irrelevant features, reduce information loss during downsampling, further improve the recognition accuracy, and effectively perform sign language recognition.

Description

Technical Field

[0001] This invention relates to the field of sign language recognition technology, and in particular to a sign language recognition method based on 3D-ResNet. Background Technology

[0002] Sign language is an important tool for communication for deaf and mute people. Research on sign language recognition is of great significance for improving their quality of life and also provides insights for research in human-computer interaction fields such as smart home control and traffic gesture recognition. Due to the spatiotemporal redundancy and weak spatiotemporal correlation in sign language videos, sign language recognition is a very challenging task. Based on different technologies, sign language recognition can be divided into traditional methods and deep learning-based methods. Traditional methods require manual feature extraction, which is very time-consuming. In recent years, deep learning has developed rapidly, showing significant advantages in image classification, object detection, and recognition. Furthermore, influenced by the current era of big data, deep learning-based sign language recognition methods are becoming the mainstream research trend. When processing video data, temporal correlation needs to be considered. Long Short-Term Memory (LSTM) networks are used by some researchers to process video data because they can effectively solve long-term dependency problems. Liu et al. proposed an end-to-end sign language recognition method based on LSTM, using the motion trajectories of four skeletal joints as input data. Because video data contains spatiotemporal information, 3D-Convolutional Neural Networks (3D-CNNs) have emerged, capable of extracting the correlation information between video frames while acquiring spatial information. Wang Fenghua et al. improved the I3D network model structure, proposing a sign language recognition method, CBAM-I3D, which integrates dual-stream 3D-CNNs and an attention mechanism, achieving a recognition rate of 90.76%. Zhou et al. first used the Laplacian operator to extract high-quality image sequences, then fed them into a designed (3+2+1)D residual network for sign language recognition, achieving good recognition results on a sign language dataset. Xie Xiaoyan et al. proposed a dynamic gesture recognition method based on the characteristics of video data. This method uses the DBSCAN clustering algorithm to extract motion trend features from the video encoded data, then uses a random forest classifier to classify the motion trends, and finally uses a convolutional neural network to extract features for gesture recognition. The above methods have all improved upon different aspects of sign language recognition. Because 3D convolutional neural networks have significant advantages in the video domain, this invention is based on a 3D-ResNet network for sign language recognition. However, since the first large convolutional layer of 3D-ResNet is a 7×7×7 kernel, which extracts coarse-grained features, information loss occurs during the extraction process. Furthermore, the attention weights based on spatial and temporal location, as well as spatial and channel dimensions, significantly impact recognition accuracy. Therefore, this invention improves upon 3D-ResNet for sign language recognition, providing a 3D-ResNet-based sign language recognition method. Summary of the Invention

[0003] To address the above issues, the purpose of this invention is to provide a sign language recognition method based on 3D-ResNet. This method integrates the design concepts of Inception and ResNet structures, replacing the first large convolutional layer of the 3D-ResNet residual network with a 3D small convolutional residual module designed in this invention. This solves the problem of focusing more on coarse-grained features when capturing sign language action features. A quadruplet attention (QA) module designed in this invention, which can capture spatial and channel relationships, is introduced between the convolutional layers of the ResNet basic blocks. A residual connection method of pooling followed by convolution is used to reduce information loss while downsampling, further improving the recognition accuracy.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0005] A sign language recognition method based on 3D-ResNet includes the following steps:

[0006] Step 1: Extract input features from the image sequence using a 3D small convolutional residual module and a max pooling layer;

[0007] Step 2: Pass the extracted input features through four pooling convolutional residual blocks in sequence to extract high-level feature information;

[0008] Step 3: The high-level feature information obtained in Step 2 is fed into the softmax layer through the average pooling layer and the fully connected layer for classification output.

[0009] Numerous experiments have demonstrated that excessively deep network layers can actually worsen model performance, primarily due to two issues: (1) gradient vanishing, affecting network convergence; and (2) accuracy saturation. To address these problems, He et al. proposed a residual neural network, ResNet (Residual Neural Network). This network utilizes a shortcut connection approach, allowing it to skip multiple convolutional layers and directly perform identity mapping. This enables the network to learn the residual F(x) = H(x) - x. When the residual is 0, the network only performs identity mapping, and performance does not degrade. When the residual is not 0, the residual block learns new features based on the input features, thus avoiding performance degradation caused by excessive convolutional layers. The calculation of the residual block is expressed by the following formula:

[0010] y=F(x,{W i})+x(1)

[0011] Where x and y represent the input and output respectively, W i This represents the convolution operation, F(x,{W i}) represents the residual mapping that needs to be learned.

[0012] Formula (1) requires that the dimensions of x and F must be the same. When their dimensions are different, a linear mapping needs to be added to adjust the dimensions, as shown in Formula (2):

[0013] y=F(x,{W i})+W s x(2)

[0014] Among them, W s It is a convolution operation used for linear mappings.

[0015] Due to the significant advantages of 3D-CNN in the video domain, Kensho et al. combined 3D-CNN with residual neural networks to propose 3D-ResNet for action recognition. Table 1 shows the network architecture of 3D-ResNet18. Its difference from the original ResNet lies in the dimension of the convolutional kernels and pooling operations; 3D-ResNet performs 3D convolution and 3D pooling operations. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. The kernel size of conv1 is 7×7×7 with a stride of 1×2×2. Input downsampling is performed by conv3_1, conv4_1, and conv5_1, with kernel sizes of 3×3×3 and strides of 2×2×2. This invention constructs a sign language recognition model based on the aforementioned 3D-ResNet.

[0016] Table 1 3D-ResNet18 Network Architecture

[0017]

[0018] The first layer of the 3D-ResNet network is a 7×7×7 convolutional layer. This large 7×7×7 kernel extracts coarse-grained features, while the fine-grained features of hand gestures in sign language videos are crucial for sign language recognition. The Inception module improves network performance by stacking multiple small convolutional layers to increase network width. Since shallow networks have high-resolution but low-level semantic features, while deep networks have low-resolution but high-level semantic features, the ResNet module uses a jumper structure to connect different layers to achieve feature fusion of different resolutions. This invention combines the ideas of the Inception and ResNet modules to design a 3D small convolutional residual module to replace the large first convolutional layer of the 3D-ResNet network.

[0019] Preferably, the 3D small convolutional residual module in step 1 is a convolutional module designed by combining the ideas of the Inception module and the ResNet module. The 3D small convolutional residual module includes two main branches and one shortcut branch. One main branch includes one convolutional layer, and the other main branch includes two convolutional layers. The kernel size of the three convolutional layers in the two main branches is 3×3×3. The number of channels in the convolutional layer of the first main branch is 32, and the number of channels in the two convolutional layers of the second main branch are 16 and 32, respectively. The shortcut branch includes one convolutional layer, and each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. The outputs of the two main branches are concatenated in the channel dimension. In order for the network to fully extract the input features, this invention introduces residual connections, adding the concatenation result of the two main branches to the output of the shortcut branch, so that the network performs residual learning with reference to the input features, thereby achieving effective feature extraction of the input data.

[0020] Preferably, the four pooling convolutional residual blocks in step 2 sequentially include attention residual block one (improved residual block 1) and three attention residual blocks two (improved residual blocks 2, 3, and 4). Attention residual block one includes two cascaded improved ResNet base blocks one. The improved ResNet base blocks one has a QA module cascaded between the first ReLU activation function and the second convolutional layer of the ResNet base blocks one. Attention residual block two includes the improved ResNet base blocks two and the improved ResNet base blocks one cascaded together. The shortcut branch of the improved ResNet base blocks two has a max pooling layer with a stride of 2×2×2 before the convolutional layer, and the stride of the convolutional layer of the shortcut branch is 1×1×1. The main branch of the improved ResNet base blocks two has a QA module cascaded between the first ReLU activation function and the second convolutional layer.

[0021] ResNet has two types of residual base blocks: ResNet Base Block 1 and ResNet Base Block 2. When the input and output feature sizes are the same, they are directly connected to form ResNet Base Block 1. When the input and output feature sizes are different, residual connections are used to form ResNet Base Block 2. The shortcut branch of the residual base block has a convolutional layer stride of 2×2×2, which is used to halve the input feature map size to achieve the same size as the output. However, directly using a convolutional layer with a stride of 2×2×2, although achieving downsampling, weakens fine-grained features, causing a certain degree of distortion in the feature map, thus reducing the effective features that can be extracted by subsequent residual modules. Therefore, this invention improves ResNet Base Block 1 and ResNet Base Block 2. The improvement to ResNet Base Block 1 is that the QA module is placed between the first ReLU activation function and the second convolutional layer of ResNet Base Block 1. The improvement to ResNet's base block two is as follows: First, a max-pooling layer with a stride of 2×2×2 is added. Then, the stride of the 1×1×1 convolutional layer is changed from 2×2×2 to 1×1×1 and placed after the max-pooling layer, preserving hand details while downsampling. Based on this, the QA module is placed between the first ReLU activation function and the second convolutional layer of ResNet's base block two. The two improved ResNet base blocks are concatenated to form the attention residual block one in this invention. The improved ResNet base block two and the improved ResNet base block one are concatenated to form the attention residual block two of the model in this invention. This allows important information to be preserved for subsequent deep feature extraction, reducing information loss while downsampling, further improving recognition accuracy, and enabling effective sign language recognition.

[0022] In sign language recognition, hand and arm movements are key areas of focus, and determining their spatial and temporal locations is beneficial for improving recognition rates. Common attention mechanisms in computer vision include SE and CBAM. SE only focuses on channel attention, while CBAM, although applying attention both channel-wise and spatially, does not consider their interaction relationships. Triplet Attention introduces an interaction mechanism, using rotation operations to establish the interaction relationship between the channel and spatial dimensions. It consists of three branches: the first two branches calculate the attention weights between the channel and spatial dimensions, and the third branch calculates the spatial attention weights. Triplet Attention is a 2D convolutional attention mechanism; this invention borrows its ideas to design a QA (Quadruplet Attention) module applicable to 3D convolution, simultaneously analyzing spatial and temporal features, strengthening the weights of useful features, and weakening the influence of irrelevant features.

[0023] Preferably, the QA module is a QA module applied to three-dimensional convolution, for input tensor x∈R C×T×H×W Send them to the following four branches respectively:

[0024] Branch 1: Used to calculate the attention weights of the channel dimension C and the spatial dimension (H,W). The input first undergoes cross-dimensional interaction to establish (C,H,W) dimensional interaction, then passes through a Z-Pool and a k×k×k three-dimensional convolutional layer, and then obtains the attention weights through the Sigmoid activation function. Finally, it undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input. The Z-Pool refers to concatenating the max pooling result and the average pooling result. Its mathematical formula is expressed by equation (3):

[0025] Z-pool(x) = [MaxPool 0d (x),AvgPool 0d (x)](3)

[0026] Where x represents the input, and 0d is the 0th dimension of the tensor when performing max pooling and average pooling respectively;

[0027] Branch 2: Used to calculate the attention weights of the channel dimension C and the spatial dimension (T,W). The input first undergoes cross-dimensional interaction to establish (C,T,W) dimensional interaction, then passes through a Z-Pool and a k×k×k three-dimensional convolutional layer, then obtains the attention weights through the Sigmoid activation function, and finally undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input.

[0028] Branch 3: Responsible for capturing the interaction between channel dimension C and spatial dimension (T,H). The input first undergoes cross-dimensional interaction to establish (C,T,H) dimensional interaction, then passes through Z-Pool and k ×k×k three-dimensional convolutional layers, then obtains attention weights through the Sigmoid activation function, and finally undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input.

[0029] Branch 4: Used to calculate attention weights in spatial dimensions (T,H,W). There are no cross-dimensional interaction operations. The input is directly fed into the Z-Pool and a k×k×k 3D convolutional layer, and then the attention weights can be obtained by passing the Sigmoid activation function.

[0030] Finally, averaging the attention weights of the four branches yields the output of the QA module:

[0031]

[0032] Where x represents the input, and These represent the cross-dimensional interaction operations of the first three branches. and This represents the Z-Pool operation, where ψ1, ψ2, ψ3, and ψ4 represent the convolution operations in the four branches, and σ represents the Sigmoid activation function. and This represents the reverse cross-dimensional operation of the first three branches.

[0033] Simplifying formula (4) yields:

[0034]

[0035] Where ω1, ω2, ω3, and ω4 represent the attention weights of the four branches, respectively. and This indicates a reverse cross-dimensional operation that maintains the same shape as the original input tensor.

[0036] Table 2 is a comparison table of attention levels.

[0037]

[0038] As shown in Table 2, the performance of both SE and CBAM decreased to varying degrees when attention was added. The SE module only learned the correlation between channels, ignoring spatial relationships, resulting in an accuracy of only 87.9%. While CBAM combines channel and spatial attention, they are performed separately. Channel attention is used first to weight the feature maps, disrupting the temporal order of the input features and affecting the effectiveness of subsequent spatial attention. Compared to 3D-ResNet without attention, its accuracy decreased by 4.5%, the largest drop. However, QA not only considers channel and spatial attention but also captures the interaction between channels and space, enabling the model to achieve an accuracy of 91.0%.

[0039] Furthermore, other architectures or settings of the sign language recognition method based on 3D-ResNet of the present invention all adopt conventional settings in the art that enable it to function properly. For example, SE and CBAM, softmax function, max pooling layer, average pooling layer, fully connected layer, Inception module, ResNet module, etc. are all commonly used technical means in the art and will not be described in detail here.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] This invention addresses the issue of focusing more on group-level features when capturing sign language action features by replacing the first large convolutional layer of the 3D-ResNet network with a 3D small convolutional residual module, thereby extracting more effective low-level features. A Quadruplet Attention (QA) module is introduced between the convolutional layers of the two residual base blocks of ResNet to capture spatial and channel relationships, simultaneously analyzing spatial and temporal features, strengthening the weight of useful features and weakening the influence of irrelevant features. Furthermore, a residual connection method using pooling followed by convolution is used to improve the shortcut branch of the second base block of ResNet, allowing important information to be retained for subsequent deep feature extraction. This reduces information loss during downsampling, further improving recognition accuracy and enabling effective sign language recognition. Attached Figure Description

[0042] Figure 1 This is an overall structural diagram of a sign language recognition method based on 3D-ResNet according to the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the first large convolutional layer in a 3D-ResNet network.

[0044] Figure 3 This is a structural diagram of the QA module;

[0045] Figure 4 This is a schematic diagram of the structure of two types of residual basic blocks in ResNet;

[0046] Figure 5 This is a schematic diagram of the structures of attention residual block one and attention residual block two;

[0047] Figure 6 The images shown in the example are the original image (a) and the preprocessed image (b).

[0048] Figure 7 The validation accuracy and loss curves of the improved 3D-ResNet and the original 3D-ResNet in the examples are shown.

[0049] Figure 8 The examples show the heatmaps of the original 3D-ResNet model on four sign language videos and the corresponding heatmaps of the videos using the improved 3D-ResNet model.

[0050] Figure 9 The examples show the validation accuracy and loss curves of the original 3D-ResNet model, the convolutional module based on Inception and ResNet, the 3D-ResNet model using the QA module, and the pooled convolutional residual block, respectively.

[0051] Figure 10 The image shows a heatmap of the large convolutional layer and the convolutional modules based on Inception and ResNet in the embodiment.

[0052] Figure 11 The image shows a heatmap of the ResNet residual base block and the residual base block after adding the QA module in the embodiment.

[0053] Figure 12 The image shown is a heatmap of the results after processing with convolutional residual blocks and pooling convolutional residual blocks in the embodiment. Detailed Implementation

[0054] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0055] Example

[0056] The dataset used is the CSL dataset from the University of Science and Technology of China, which includes a 500-class isolated vocabulary dataset and a 100-class continuous sentence dataset. Both datasets contain RGB video, depth video, and skeletal joint data. In this embodiment, 100 classes of vocabulary were selected from the 500-class isolated vocabulary dataset, and experiments were conducted using only RGB video data, which was divided into training, validation, and test sets in a 7:2:1 ratio. Sixteen frames were extracted from each video using uniform sampling. Since the original video resolution was 1280×720, the file size was too large, and the original images contained a large amount of background-irrelevant information, such as... Figure 6 As shown in (a), in this embodiment, a region with a width and height of 600 is cropped centered on the sign language interpreter in the video. Then, the resize function is used to preprocess each frame to a size of 224×224 as the input to the neural network. Figure 6 As shown in (b). The final experimental data consisted of 25,000 videos and 400,000 images, including 17,500 videos in the training set, 5,000 videos in the validation set, and 2,500 videos in the test set.

[0057] like Figure 1 As shown, a sign language recognition method based on 3D-ResNet includes the following steps:

[0058] Step 1: Extract input features from the selected video image sequence using a 3D mini-convolutional residual module and a max-pooling layer. The 3D mini-convolutional residual module is a convolutional module designed by combining the ideas of the Inception and ResNet modules. It includes two main branches and one shortcut branch. One main branch contains one convolutional layer, and the other contains two convolutional layers. The kernel size of the three convolutional layers in both main branches is 3×3×3. The first main branch has 32 channels, and the two convolutional layers in the second main branch have 16 and 32 channels respectively. The shortcut branch contains one convolutional layer, and each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. The outputs of the two main branches are concatenated along the channel dimension, and the concatenated result is added to the output of the shortcut branch, thereby achieving effective feature extraction from the input data.

[0059] Step 2: The extracted input features are sequentially passed through four pooling convolutional residual blocks to extract high-level feature information. These four pooling convolutional residual blocks sequentially include attention residual block one (improved residual block 1) and three attention residual blocks two (improved residual blocks 2, 3, and 4). Attention residual block one includes two cascaded improved ResNet base blocks. A schematic diagram of the first large convolutional layer of the 3D-ResNet network is shown below. Figure 2 As shown, the structural diagrams of the two types of residual basic blocks in ResNet are as follows: Figure 4 As shown, the improved ResNet's base block one has a QA module cascaded between the first ReLU activation function and the second convolutional layer of the base block one; the attention residual block two includes the improved ResNet base block two and the improved ResNet base block one cascaded together. The shortcut branch of the improved ResNet base block two has a max pooling layer with a stride of 2×2×2 before its convolutional layer, and the shortcut branch's convolutional layer has a stride of 1×1×1. A QA module is cascaded between the first ReLU activation function and the second convolutional layer of the main branch of the improved ResNet base block two. The structural diagrams of attention residual block one and attention residual block two are shown below. Figure 5 As shown.

[0060] The QA module is a QA module applied to 3D convolution, and its structure is as follows: Figure 3 As shown, for the input tensor x∈R C ×T×H×W Send them to the following four branches respectively:

[0061] Branch 1: Used to calculate the attention weights of the channel dimension C and the spatial dimension (H,W). The input first undergoes cross-dimensional interaction to establish (C,H,W) dimensional interaction, then passes through a Z-Pool and a k×k×k three-dimensional convolutional layer, and then obtains the attention weights through the Sigmoid activation function. Finally, it undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input. The Z-Pool refers to concatenating the max pooling result and the average pooling result. Its mathematical formula is expressed by equation (3):

[0062] Z-pool(x) = [MaxPool 0d (x),AvgPool 0d (x)](3)

[0063] Where x represents the input, and 0d is the 0th dimension of the tensor when performing max pooling and average pooling respectively;

[0064] Branch 2: Used to calculate the attention weights of the channel dimension C and the spatial dimension (T,W). The input first undergoes cross-dimensional interaction to establish (C,T,W) dimensional interaction, then passes through a Z-Pool and a k×k×k three-dimensional convolutional layer, then obtains the attention weights through the Sigmoid activation function, and finally undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input.

[0065] Branch 3: Responsible for capturing the interaction between channel dimension C and spatial dimension (T,H). The input first undergoes cross-dimensional interaction to establish (C,T,H) dimensional interaction, then passes through Z-Pool and k ×k×k three-dimensional convolutional layers, then obtains attention weights through the Sigmoid activation function, and finally undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input.

[0066] Branch 4: Used to calculate attention weights in spatial dimensions (T,H,W). There are no cross-dimensional interaction operations. The input is directly fed into the Z-Pool and a k×k×k 3D convolutional layer, and then the attention weights can be obtained by passing the Sigmoid activation function.

[0067] Finally, averaging the attention weights of the four branches yields the output of the QA module:

[0068]

[0069] in x Represents input, and These represent the cross-dimensional interaction operations of the first three branches. and This represents the Z-Pool operation, where ψ1, ψ2, ψ3, and ψ4 represent the convolution operations in the four branches, and σ represents the Sigmoid activation function. and Represents the reverse cross-dimensional operation of the first three branches;

[0070] Simplifying formula (4) yields:

[0071]

[0072] Where ω1, ω2, ω3, and ω4 represent the attention weights of the four branches, respectively. and This indicates a reverse cross-dimensional operation that maintains the same shape as the original input tensor.

[0073] Step 3: The high-level feature information obtained in Step 2 is fed into the softmax layer through the average pooling layer and the fully connected layer for classification output.

[0074] Comparative example:

[0075] The experimental hardware configuration is as follows: Core TM The system consists of an i9-9900KF CPU, 32GB of RAM, and two NVIDIA GeForce RTX 2080 Ti graphics cards. The software environment is Windows 10 operating system, CUDA version 10.0, PyTorch version 1.6.0, and Python version 3.7.9. Model training employs the ADMA optimization algorithm and exponential decay learning rate, with gamma set to 0.9, an initial learning rate of 0.0001, a batch size of 4, and 50 epochs.

[0076] like Figure 7 As shown, the validation accuracy and loss curves of the improved 3D-ResNet and the original 3D-ResNet are presented in this paper. The dashed line represents the original 3D-ResNet model, and the solid line represents the improved 3D-ResNet model. It can be clearly seen that the accuracy of the improved 3D-ResNet increases faster, reaches a stable value earlier than the original 3D-ResNet, and the final value is higher than that of the original 3D-ResNet. The loss reaches a smaller stable value faster than that of the original 3D-ResNet.

[0077] like Figure 8As shown, the first row is the heatmap of the original 3D-ResNet model on four sign language videos, and the second row is the heatmap of the improved 3D-ResNet model of this invention on the corresponding videos. In the heatmap, the white area and areas within the white area represent the areas the network pays attention to, while the areas outside the white area represent the areas the network does not pay attention to. It can be seen from the graph that the original 3D-ResNet model's focus area is away from the sign language interpreter, while the improved 3D-ResNet model's focus area is entirely on the sign language interpreter. Although the focus within the white area is still slightly off-center from the arms, the arms are darker in color than in the original model, indicating a greater attention weight.

[0078] To verify the effectiveness of the proposed model, it was compared with other sign language recognition models on the CSL dataset, and the results are shown in Table 3. The HMM-based method requires manual extraction of effective features, which is difficult, computationally complex, and has the lowest accuracy. Compared with traditional methods, deep learning-based methods perform better and can meet the requirements of sign language recognition to a certain extent. Among these methods, the improved 3D-ResNet model proposed in this invention achieves the highest accuracy, reaching 91.9%, and the experimental results strongly demonstrate the effectiveness of the proposed method.

[0079] Table 3 Comparison of experimental results of our proposed method with other methods on the CSL dataset.

[0080]

[0081] Ablation experiment:

[0082] To further illustrate the contribution of each improvement method in this invention, ablation experiments were conducted. For example... Figure 9 As shown, the graph represents the validation accuracy and loss of the original 3D-ResNet and models using convolutional modules based on Inception and ResNet, a 3D-ResNet model using a QA module, and a 3D-ResNet model using pooled convolutional residual blocks, respectively. The horizontal axis represents the training epoch, and the vertical axis represents the accuracy or loss. The dashed line represents the original 3D-ResNet model, and the solid lines represent models using the three improved methods. Figure 9(a) shows the validation accuracy curves of the original 3D-ResNet model with large convolutional layers and convolutional modules based on Inception and ResNet; (b) shows the validation loss curves of both; (c) and (d) show the accuracy and loss of the 3D-ResNet model before and after adding the QA module, respectively; (e) and (f) show the accuracy and loss curves of the convolutional residual blocks and pooling convolutional residual blocks of the original 3D-ResNet model, respectively. As can be seen from the curves, the validation accuracy curves of the three improved methods rise faster than those of the original 3D-ResNet, and the final stable accuracy is higher than that of the original 3D-ResNet to varying degrees. The pooling convolutional residual block shows the largest improvement, followed by the 3D-ResNet with added QA modules. The loss of all three improved methods decreases faster than that of the original 3D-ResNet, and the final loss value is also lower, with smoother curves. The effect of adding QA modules to 3D-ResNet is more significant.

[0083] like Figure 10-12 As shown, heatmaps are displayed for the corresponding layers using the original 3D-ResNet, the convolutional modules based on Inception and ResNet, the 3D-ResNet model using the QA module, and the pooled convolutional residual blocks, respectively. Figure 10 The heatmap based on Inception and ResNet convolutional modules shows that, compared to the large convolutional layers of the original 3D-ResNet, the white areas of the sign language interpreter are brighter and more focused on the hands, but the difference is not very obvious. In Figure 11, the 3D-ResNet model using the QA module, with the added attention residual blocks 1, 2, 3 and 4, all pay more attention to the sign language interpreter's hands and arms compared to before the improvement. Although the attention area is still shifted after the improvement, the white areas of the hands are brighter and the attention weight is increased. Figure 12 In the process, pooling and convolution of residual blocks, and the operation of pooling followed by convolution in the shortcut branch of residual blocks, ensures that important information is preserved during feature map downsampling, and the heatmap shows that more attention is paid to the sign language user's hand movements.

[0084] The experimental results of the above ablation experiments are shown in Table 4.

[0085] Table 4 Ablation Experiment Results

[0086]

[0087] This invention proposes a sign language recognition method based on 3D-ResNet, utilizing a QA module and an improved 3D-ResNet network. Since fine-grained hand features are crucial information in sign language videos, this invention replaces the large convolutional layers of 3D-ResNet with small 3D convolutional residual modules, thereby extracting more effective low-level features. Simultaneously, to ensure the model pays attention to the hand region, a QA module capable of cross-dimensional interactive attention weighting calculations across channel and spatial dimensions is incorporated, simultaneously analyzing spatial and temporal features, strengthening the weight of useful features, and weakening the influence of irrelevant features. Furthermore, to address the severe information loss during residual block downsampling, the shortcut branch of the residual block is improved, allowing important information to be preserved for subsequent deep feature extraction. On the CSL dataset, the proposed QA-based and improved 3D-ResNet sign language recognition model achieves higher accuracy than the original 3D-ResNet and other sign language recognition algorithms, effectively enabling sign language recognition.

[0088] The embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A sign language recognition method based on 3D-ResNet, characterized in that, Includes the following steps: Step 1: Extract input features from the image sequence using a 3D small convolutional residual module and a max pooling layer; Step 2: Pass the extracted input features through four pooling convolutional residual blocks in sequence to extract high-level feature information; Step 3: The high-level feature information obtained in Step 2 is fed into the softmax layer through the average pooling layer and the fully connected layer for classification output; The 3D mini-convolutional residual module in step 1 is a convolutional module designed by combining the ideas of the Inception module and the ResNet module. The 3D mini-convolutional residual module includes two main branches and one shortcut branch. One main branch includes one convolutional layer, and the other main branch includes two convolutional layers. The kernel size of the three convolutional layers in the two main branches is 3×3×3. The number of channels in the convolutional layer of the first main branch is 32, and the number of channels in the two convolutional layers of the second main branch are 16 and 32 respectively. The shortcut branch includes one convolutional layer. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. The outputs of the two main branches are concatenated in the channel dimension. The concatenation result of the two main branches is added to the output of the shortcut branch, thereby realizing effective feature extraction of the input data.

2. The sign language recognition method based on 3D-ResNet according to claim 1 is characterized in that, The four pooling convolutional residual blocks in step 2 sequentially include attention residual block 1 and three attention residual blocks 2. Attention residual block 1 includes two cascaded improved ResNet base blocks 1. The improved ResNet base blocks 1 have a quadruple attention QA module cascaded between the first ReLU activation function and the second convolutional layer of the ResNet base blocks 1. Attention residual block 2 includes the improved ResNet base blocks 2 and the improved ResNet base blocks 1 cascaded together. The shortcut branch of the improved ResNet base blocks 2 has a max pooling layer with a stride of 2×2×2 before the convolutional layer, and the stride of the convolutional layer of the shortcut branch is 1×1×1. The main branch of the improved ResNet base blocks 2 has a quadruple attention QA module cascaded between the first ReLU activation function and the second convolutional layer.

3. The sign language recognition method based on 3D-ResNet according to claim 2 is characterized in that, The quadruple attention QA module is a quadruple attention QA module applied to 3D convolution, for input tensor x∈R C×T×H×W Send them to the following four branches respectively: Branch 1: Used to calculate the attention weights of the channel dimension C and the spatial dimension (H,W). The input first undergoes cross-dimensional interaction to establish (C,H,W) dimensional interaction, then passes through a Z-Pool and a k×k×k three-dimensional convolutional layer, and then obtains the attention weights through the Sigmoid activation function. Finally, it undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input. The Z-Pool refers to concatenating the max pooling result and the average pooling result. Its mathematical formula is expressed by equation (3): (3) in Indicates input, It is the 0th dimension of the tensor when performing max pooling and average pooling respectively; Branch 2: Used to calculate the attention weights of the channel dimension C and the spatial dimension (T,W). The input first undergoes cross-dimensional interaction to establish (C,T,W) dimensional interaction, then passes through a Z-Pool and a k×k×k three-dimensional convolutional layer, then obtains the attention weights through the Sigmoid activation function, and finally undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input. Branch 3: Responsible for capturing the interaction between channel dimension C and spatial dimension (T,H). The input first undergoes cross-dimensional interaction to establish (C,T,H) dimensional interaction, then passes through Z-Pool and k×k×k three-dimensional convolutional layers, then obtains attention weights through the Sigmoid activation function, and finally undergoes reverse cross-dimensional interaction to obtain a tensor with the same shape as the original input. Branch 4: Used to calculate attention weights in spatial dimensions (T,H,W). There are no cross-dimensional interaction operations. The input is directly fed into the Z-Pool and a k×k×k 3D convolutional layer, and then the attention weights can be obtained by passing the Sigmoid activation function. Finally, averaging the attention weights of the four branches yields the output of the four-fold attention QA module: (4) in Represents input, , and These represent the cross-dimensional interaction operations of the first three branches. , , and This indicates a Z-Pool operation. , , and These represent the convolution operations in the four branches. Represents the Sigmoid activation function. , and Represents the reverse cross-dimensional operation of the first three branches; Simplifying formula (4) yields: (5) in , , and These represent the attention weights for the four branches, , and This indicates a reverse cross-dimensional operation that maintains the same shape as the original input tensor.

Images