Continuous sign language recognition method based on action and time sequence adaptive enhancement
By using the action-adaptive enhancement module and the temporal-adaptive enhancement module, the problems of limited receptive field and computational burden in the existing technology are solved, achieving efficient continuous sign language recognition and improving detection accuracy and robustness.
Patent Information
- Application Number
- CN202510983943.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-17
- Publication Date
- 2025-10-31
AI Technical Summary
Existing continuous sign language recognition methods have limited receptive fields due to fixed structures, making it difficult to adaptively associate with distant spatial regions, and relying on external human key point supervision leads to additional computational burden.
By employing an action-adaptive enhancement module and a temporal-adaptive enhancement module, unsupervised inter-frame trajectory modeling is achieved through multi-scale feature extraction and cascaded temporal convolution. Combined with end-to-end training, detection accuracy is improved.
Without increasing computational overhead, the receptive field in the time dimension is effectively expanded, improving the model's ability to understand dynamic changes in sign language and increasing the accuracy of continuous sign language recognition.
Smart Images
Figure CN120877375A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a continuous sign language recognition method in the field of video recognition, specifically a continuous sign language recognition method based on action and temporal adaptive enhancement. Background Technology
[0002] Continuous sign language recognition is an important task in the field of video understanding. It aims to transform a continuous sequence of video frames into a semantically coherent sequence of sign language words, which has significant application value in scenarios such as barrier-free communication and human-computer interaction.
[0003] Current mainstream methods employ a two-stage architecture. First, a shared 2D CNN is used to independently process single frames to extract spatial features. Then, a 1D CNN and a bidirectional LSTM are combined to model temporal dependencies. However, this method has significant limitations: independent frame processing leads to the loss of continuity in cross-frame human trajectory information (such as hand movement trajectories and facial expressions), weakening action discriminative power. To introduce spatiotemporal interaction, mainstream solutions attempt to use 3D convolution, (2+1)D convolution, or temporal shift modules. However, their fixed receptive fields are limited, making it difficult to adaptively associate distant spatial regions (such as the position of rapidly moving hands). Furthermore, existing enhancement schemes rely on external human keypoint supervision, introducing additional computational burden. Summary of the Invention
[0004] To address the limitations of existing methods that rely on fixed receptive fields and require additional supervisory cues, this invention proposes a continuous sign language recognition method based on action and temporal adaptive enhancement. This method effectively improves the accuracy of continuous sign language recognition without incurring additional overhead, thus reducing the cost of end-to-end training.
[0005] The technical solution adopted in this invention is as follows: it includes the following steps:
[0006] I. A continuous sign language recognition method based on action and temporal adaptive enhancement
[0007] 1) After preprocessing the video data, a training set is obtained;
[0008] 2) Construct a neural network model for continuous sign language recognition. The neural network model includes a feature extractor, an action adaptive enhancement module, a temporal adaptive enhancement module, a temporal modeling module, and a classification head. The feature extractor is connected to the action adaptive enhancement module and the temporal adaptive enhancement module. Both the action adaptive enhancement module and the temporal adaptive enhancement module are connected to the temporal modeling module, and the temporal modeling module is connected to the classification head. Then, train the neural network model using the training set to obtain a trained neural network model.
[0009] 3) Input the image sequence to be detected into the trained neural network model, and the model outputs the continuous sign language recognition result.
[0010] The preprocessing includes image scaling, cropping, horizontal flipping, and adding random perturbations.
[0011] The action-adaptive enhancement module includes a downsampling projection layer, a convolutional layer, an upsampling projection layer, and an activation layer. The input to the action-adaptive enhancement module is used as the input to the downsampling projection layer, which is connected to each of the four convolutional layers. The output of the downsampling projection layer is subtracted from the output of the first convolutional layer to obtain a first intermediate feature. The output of the downsampling projection layer is subtracted from the output of the second convolutional layer to obtain a second intermediate feature. The output of the downsampling projection layer is subtracted from the output of the third convolutional layer to obtain a third intermediate feature. The output of the downsampling projection layer is subtracted from the output of the fourth convolutional layer to obtain a fourth intermediate feature. The first and fourth intermediate features are multiplied by their corresponding learnable factors and then fused to obtain a fused feature vector. The output of the downsampling projection layer is subtracted from the fused feature vector to obtain a fifth intermediate feature, which is used as the input to the upsampling projection layer. The upsampling projection layer is connected to the activation layer. The input to the action-adaptive enhancement module is multiplied by the weights of the activation layer's output and then residually connected to the input of the action-adaptive enhancement module. The output of the residually connected module is used as the output of the action-adaptive enhancement module.
[0012] The first convolutional layer has a kernel size of 9×3×3, a stride of 1, and padding of (4,1,1); the second convolutional layer has a kernel size of 9×5×5, a stride of 1, and padding of (4,2,2); the third convolutional layer has a kernel size of 9×7×7, a stride of 1, and padding of (4,3,3); and the fourth convolutional layer has a kernel size of 9×9×9, a stride of 1, and padding of (4,4,4).
[0013] The temporal adaptive enhancement module includes a channel partitioning layer and multiple dilated convolutional blocks. The input of the temporal adaptive enhancement module is used as the input of the channel partitioning layer. The first output of the channel partitioning layer is used as the input of the first dilated convolutional block. The output of the first dilated convolutional block is added to the second output of the channel partitioning layer and then used as the input of the second dilated convolutional block. The output of the second dilated convolutional block is added to the third output of the channel partitioning layer and then used as the input of the third dilated convolutional block. The output of the third dilated convolutional block is added to the fourth output of the channel partitioning layer and then used as the input of the fourth dilated convolutional block. The outputs of the first and fourth dilated convolutional blocks are concatenated and then residually connected to the input of the temporal adaptive enhancement module. The output after residual connection is used as the output of the action adaptive enhancement module.
[0014] Each of the first dilated convolutional block to the fourth dilated convolutional block includes a connected SE module and a 3D convolutional layer. The dilation rates of the 3D convolutional layers in the first dilated convolutional block to the fourth dilated convolutional block are different.
[0015] In the first dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (1,1,1); in the second dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (2,1,1); in the third dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (3,1,1); in the fourth dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (4,1,1).
[0016] II. A continuous sign language recognition system based on action and temporal adaptive enhancement
[0017] An image preprocessing unit is used to preprocess the images in each image sequence to obtain a preprocessed image sequence.
[0018] Training set construction unit, used to construct a training set based on the preprocessed image sequence;
[0019] The continuous sign language recognition unit is used to store a neural network model for continuous sign language recognition and to use the neural network model to perform continuous sign language recognition on the input image sequence, thereby outputting the corresponding continuous sign language recognition result;
[0020] The training unit is used to train the neural network model using the training set.
[0021] III. A computer device
[0022] The device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the continuous sign language recognition method based on action and temporal adaptive enhancement.
[0023] IV. A computer-readable storage medium
[0024] The medium stores a computer program, which, when executed by a processor, implements the steps of the continuous sign language recognition method based on action and temporal adaptive enhancement.
[0025] V. A computer program product
[0026] The product includes a computer program / instruction that, when executed by a processor, implements the steps of the continuous sign language recognition method based on action and temporal adaptive enhancement.
[0027] The beneficial effects of this invention are:
[0028] To address the limitations of fixed receptive fields and the need for additional supervision cues, this invention proposes an action-adaptive enhancement and a timing-adaptive enhancement module. This module facilitates end-to-end training of the model while enabling unsupervised inter-frame trajectory modeling, which can effectively improve detection accuracy.
[0029] This invention employs multi-scale feature extraction in the motion adaptive enhancement module, using multiple convolutional kernels of different sizes for feature extraction. Specifically, it dynamically extracts trajectory features of different granularities through multi-branch convolution, and then adaptively fuses them to enhance important information of moving parts such as the face and hands.
[0030] In this invention, the temporal adaptive enhancement module divides the input features into multiple sub-blocks along the channel dimension, and sequentially applies cascaded temporal and spatial convolutions to some of these sub-blocks. Simultaneously, residual connections are used to achieve layer-by-layer information accumulation. This design effectively expands the receptive field along the temporal dimension without significantly increasing parameters, enabling each frame to incorporate more information from neighboring and distant frames, thereby improving the model's ability to understand dynamic changes in sign language. In other words, the temporal adaptive enhancement module proposed in this invention achieves long-term and short-term modeling, accurately modeling the spatiotemporal evolution characteristics of sign language movements while reducing computational overhead, providing an efficient solution for end-to-end continuous sign language recognition.
[0031] This invention allows for end-to-end training, where the detection head and feature extractor are combined and optimized under a single loss function, which helps the model to better express itself. Attached Figure Description
[0032] Figure 1 This refers to the neural network model for continuous sign language recognition constructed in the method of this invention.
[0033] Figure 2 This invention relates to an action adaptive enhancement module.
[0034] Figure 3 This is the timing adaptive enhancement module proposed in this invention.
[0035] Figure 4 This is a comparison diagram of the original image and the heat map in an embodiment of the present invention. Detailed Implementation
[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0037] Specific embodiments of the present invention are as follows:
[0038] The dataset used in this invention is the PHOENIX2014 continuous sign language recognition dataset. PHOENIX2014 is a German sign language dataset, collected from German weather forecast broadcasts. It has been widely used in continuous sign language recognition tasks to validate new methods and has a strong influence. This dataset includes nine different sign language demonstrators, all wearing black tops against a gray background. The video frame rate is 25fps, and all video frames have a resolution of 210×260 pixels. The dataset contains 6841 different German sentences, with a sign language vocabulary of 1295. The number of data samples in the training, validation, and test sets are 5672, 540, and 629, respectively.
[0039] This invention proposes a continuous sign language recognition method based on action and temporal adaptive enhancement, the method comprising the following steps:
[0040] 1) After preprocessing the video data, training and testing sets are obtained. Specifically, the video dataset is split into several image sequences, and each image sequence is preprocessed, including image scaling, cropping, horizontal flipping, and adding random perturbations. The preprocessing in this embodiment is as follows:
[0041] During training, the original image frames are uniformly scaled to 256×256, and then a series of data augmentation operations are applied, including random cropping to 224×224, random horizontal flipping with a 50% probability, and a 20% temporal perturbation, to improve the model's generalization ability and robustness to temporal changes. During testing, images are cropped to 224×224 using center cropping to ensure input consistency and stability.
[0042] Random pruning and random perturbation can expand the dataset size, making the model more robust to the data, reducing the risk of overfitting during training, and optimizing the model's performance on the dataset.
[0043] 2) Construct a neural network model for continuous sign language recognition, such as... Figure 1 As shown, the neural network model includes a feature extractor, an action adaptive enhancement module, a temporal adaptive enhancement module, a temporal modeling module, and a classification head. The feature extractor is connected to the action adaptive enhancement module and the temporal adaptive enhancement module. Both the action adaptive enhancement module and the temporal adaptive enhancement module are connected to the temporal modeling module, and the temporal modeling module is connected to the classification head. The neural network model is then trained using a training set to obtain a trained neural network model.
[0044] like Figure 2As shown, the action-adaptive enhancement module includes a downsampling projection layer, a convolutional layer, an upsampling projection layer, and an activation layer. The input of the action-adaptive enhancement module (i.e., the output of the feature extractor) serves as the input to the downsampling projection layer. The downsampling projection layer is connected to each of the four convolutional layers. The output of the downsampling projection layer is subtracted from the output of the first convolutional layer to obtain the first intermediate feature. The output of the downsampling projection layer is subtracted from the output of the second convolutional layer to obtain the second intermediate feature. The output of the downsampling projection layer is subtracted from the output of the third convolutional layer to obtain the third intermediate feature. The output of the downsampling projection layer is subtracted from the output of the fourth convolutional layer to obtain the third intermediate feature. The fourth intermediate feature is obtained; the first and fourth intermediate features are multiplied by their corresponding learnable factors and then fused to obtain a fused feature vector; the output of the downsampling projection layer is subtracted from the fused feature vector to obtain the fifth intermediate feature, which is used as the input to the upsampling projection layer to restore the number of channels; the upsampling projection layer is connected to an activation layer with the sigmoid activation function; the input of the action adaptive enhancement module is multiplied by the weights of the activation layer output and then residually connected to the input of the action adaptive enhancement module; the output of the residual connection is used as the output of the action adaptive enhancement module. Four convolutional layers form a multi-scale convolutional layer.
[0045] The kernel size of the first convolutional layer is 9×3×3, the stride is 1, and the padding is (4,1,1).
[0046] The kernel size of the second convolutional layer is 9×5×5, the stride is 1, and the padding is (4,2,2).
[0047] The kernel size of the third convolutional layer is 9×7×7, the stride is 1, and the padding is (4,3,3).
[0048] The kernel size of the fourth convolutional layer is 9×9×9, the stride is 1, and the padding is (4,4,4).
[0049] The action-adaptive enhancement module inputs a feature vector of shape T×C×H×W into the downsampling projection layer, resulting in a compressed feature vector with an output of T×C / r×H×W. The temporal feature vector is then input into the first through fourth convolutional layers, yielding four feature vectors of shape T×C / r×H×W. The output from the downsampling projection layer is subtracted from the outputs of the first through fourth convolutional layers, and the outputs of the four branches are multiplied by a learnable factor. Backpropagation of the gradient controls the importance of different branches, and the results are then summed and fused to obtain a fused feature vector. The output from the downsampling projection layer is subtracted from the fused feature vector again. This result is then passed through an upsampling projection layer to obtain a T×C×H×W feature vector. This upsampling projection layer's feature vector serves as the input to the activation layer. The input to the action-adaptive enhancement module is multiplied by the weights of the activation layer's output, and the result is then residually concatenated with the input to the action-adaptive enhancement module to obtain a T×C×H×W feature vector.
[0050] like Figure 3 As shown, the temporal adaptive enhancement module includes a channel partitioning layer and multiple dilated convolutional blocks. The input of the temporal adaptive enhancement module is used as the input of the channel partitioning layer, which splits the input into four parts to form a first output to a fourth output. The first output of the channel partitioning layer is used as the input of the first dilated convolutional block. The output of the first dilated convolutional block is added to the second output of the channel partitioning layer and then used as the input of the second dilated convolutional block. The output of the second dilated convolutional block is added to the third output of the channel partitioning layer and then used as the input of the third dilated convolutional block. The output of the third dilated convolutional block is added to the fourth output of the channel partitioning layer and then used as the input of the fourth dilated convolutional block. The outputs of the first and fourth dilated convolutional blocks are concatenated and then residually connected to the input of the temporal adaptive enhancement module. The output after residual connection is used as the output of the action adaptive enhancement module. The concatenation operation is used to restore the number of channels. The temporal adaptive enhancement module proposed in this invention can effectively expand the receptive field in the temporal dimension without significantly increasing the parameters, enabling each frame to fuse more information from neighboring and distant frames, thereby improving the model's ability to understand dynamic changes in sign language.
[0051] The first to fourth dilated convolutional blocks each include connected SE modules and 3D convolutional layers. The dilation rates of the 3D convolutional layers in the first to fourth dilated convolutional blocks are different.
[0052] In the first dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (1,1,1).
[0053] In the second dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (2,1,1).
[0054] In the third dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (3,1,1).
[0055] In the fourth dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (4,1,1).
[0056] The temporal adaptive enhancement module divides the input channel into layers using a T×C×H×W feature vector. The input features are divided into four sub-blocks along the channel dimension, each with a feature vector of T×C / 4×H×W. Concatenated temporal and spatial convolutions (i.e., dilated convolutional blocks) are applied sequentially to some of these sub-blocks, while residual connections facilitate layer-by-layer information accumulation. Finally, the feature vectors of the four dilated convolutional blocks are concatenated to obtain an output vector of T×C×H×W. This design effectively expands the receptive field in the temporal dimension without significantly increasing the number of parameters, allowing each frame to incorporate more information from neighboring and distant frames, thereby improving the model's ability to understand dynamic changes in sign language.
[0057] The feature extractor in this invention can be any backbone network for video understanding. This invention uses ResNet18 from 2D CNNs as the feature extractor for illustration.
[0058] The temporal modeling module in this invention can be any temporal modeling module for video understanding. This invention uses 1D CNN and bidirectional LSTM as temporal modeling modules for illustration.
[0059] The classification head used in this invention can be any classification head for continuous sign language recognition. This invention uses a fully connected layer as the classification head for illustration. This classification head is used to map frame-level features to a certain gloss (i.e., classification) in the sign language vocabulary.
[0060] The classification head used in this invention can be any continuous sign language recognition classification head. This invention uses CTC loss as the loss function for explanation.
[0061] 3) Input the image sequence to be detected into the trained neural network model, and the model outputs the continuous sign language recognition result, that is, the video sequence to be detected in the input video sequence.
[0062] The test set from step 1) is input into the trained neural network model from step 2) for testing, and evaluation metrics are output. The main evaluation metric used is word error rate (WER). Table 1 shows the performance of this invention in ablation experiments on the PHOENIX2014 dataset. Table 2 compares the performance of this invention with other methods on the PHOENIX2014 dataset. From Table 1, it is first noted that both action-adaptive augmentation and temporal-adaptive augmentation modules can bring significant accuracy improvements, with accuracies of 19.6% and 19.8% on the validation set and 19.1% and 19.4% on the test set. When the two modules are combined, the effectiveness is further activated, with accuracies of 18.4% and 18.9% on the validation set and test set, respectively. Table 2 shows that the method of this invention outperforms the CorrNet method.
[0063] Table 1 shows the ablation experiments of this invention on the PHOENIX2014 dataset.
[0064] motion adaptive enhancement Timing Adaptive Enhancement Validation set (%) Test set (%) - - 20.2 21 √ 19.1 19.4 √ 19.6 19.8 √ √ 18.4 18.9
[0065] Table 2 shows the performance comparison of the present invention with other methods on the PHOENIX2014 dataset.
[0066] Validation set (%) Test set (%) CorrNet 18.8 19.4 Method of the present invention 18.4 18.9
[0067] Figure 4 The generated heatmap is shown. It can be seen that the model can focus on human body regions (light yellow areas), especially showing stronger attention to key areas such as the hands and face (dark red areas). This indicates that the proposed module can dynamically highlight important regions expressing sign language meaning (such as hands and face) while suppressing attention to irrelevant regions. This demonstrates that the method of this invention effectively guides the visual module to accurately locate key visual regions. This advantage stems from the module's ability to enhance the extraction of visual information related to sign language expression and its enhanced temporal modeling capabilities when processing sign language videos, thereby improving prediction performance.
[0068] This invention proposes a continuous sign language recognition system based on action and temporal adaptive enhancement, comprising:
[0069] An image preprocessing unit is used to preprocess the images in each image sequence to obtain a preprocessed image sequence.
[0070] Training set construction unit, used to construct a training set based on the preprocessed image sequence;
[0071] The continuous sign language recognition unit is used to store a neural network model for continuous sign language recognition and to use the neural network model to perform continuous sign language recognition on the input image sequence, thereby outputting the corresponding continuous sign language recognition result;
[0072] The training unit is used to train the neural network model using the training set.
[0073] This invention proposes a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of a continuous sign language recognition method based on action and temporal adaptive enhancement.
[0074] This invention proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a continuous sign language recognition method based on action and timing adaptive enhancement.
[0075] This invention proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a continuous sign language recognition method based on action and timing adaptive enhancement.
[0076] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.
Claims
1. A continuous sign language recognition method based on action and temporal adaptive enhancement, characterized in that, Includes the following steps: 1) After preprocessing the video data, a training set is obtained; 2) Construct a neural network model for continuous sign language recognition. The neural network model includes a feature extractor, an action adaptive enhancement module, a temporal adaptive enhancement module, a temporal modeling module, and a classification head. The feature extractor is connected to the action adaptive enhancement module and the temporal adaptive enhancement module. Both the action adaptive enhancement module and the temporal adaptive enhancement module are connected to the temporal modeling module, and the temporal modeling module is connected to the classification head. Then, train the neural network model using the training set to obtain a trained neural network model. 3) Input the image sequence to be detected into the trained neural network model, and the model outputs the continuous sign language recognition result.
2. The continuous sign language recognition method based on action and temporal adaptive enhancement according to claim 1, characterized in that, The preprocessing includes image scaling, cropping, horizontal flipping, and adding random perturbations.
3. The continuous sign language recognition method based on action and temporal adaptive enhancement according to claim 1, characterized in that, The action adaptive enhancement module includes a downsampling projection layer, a convolutional layer, an upsampling projection layer, and an activation layer. The input of the action adaptive enhancement module is used as the input of the downsampling projection layer. The downsampling projection layer is connected to the four convolutional layers respectively. The output of the downsampling projection layer is subtracted from the output of the first convolutional layer to obtain the first intermediate feature. The second intermediate feature is obtained by subtracting the output of the downsampling projection layer from the output of the second convolutional layer. The output of the downsampling projection layer is subtracted from the output of the third convolutional layer to obtain the third intermediate feature; the output of the downsampling projection layer is subtracted from the output of the fourth convolutional layer to obtain the fourth intermediate feature; the first and fourth intermediate features are multiplied by their corresponding learnable factors and then added together to obtain a fused feature vector; the output of the downsampling projection layer is subtracted from the fused feature vector to obtain the fifth intermediate feature, which is then used as the input to the upsampling projection layer. The upsampling projection layer is connected to the activation layer. The input of the action adaptive enhancement module is multiplied by the weights of the output of the activation layer and then residually connected to the input of the action adaptive enhancement module. The output after residual connection is used as the output of the action adaptive enhancement module.
4. The continuous sign language recognition method based on action and temporal adaptive enhancement according to claim 3, characterized in that, The first convolutional layer has a kernel size of 9×3×3, a stride of 1, and padding of (4,1,1); the second convolutional layer has a kernel size of 9×5×5, a stride of 1, and padding of (4,2,2); the third convolutional layer has a kernel size of 9×7×7, a stride of 1, and padding of (4,3,3); and the fourth convolutional layer has a kernel size of 9×9×9, a stride of 1, and padding of (4,4,4).
5. The continuous sign language recognition method based on action and temporal adaptive enhancement according to claim 1, characterized in that, The temporal adaptive enhancement module includes a channel partitioning layer and multiple dilated convolutional blocks. The input of the temporal adaptive enhancement module is used as the input of the channel partitioning layer. The first output of the channel partitioning layer is used as the input of the first dilated convolutional block. The output of the first dilated convolutional block is added to the second output of the channel partitioning layer and then used as the input of the second dilated convolutional block. The output of the second dilated convolutional block is added to the third output of the channel partitioning layer and then used as the input of the third dilated convolutional block. The output of the third dilated convolutional block is added to the fourth output of the channel partitioning layer and then used as the input of the fourth dilated convolutional block. The outputs of the first and fourth dilated convolutional blocks are concatenated and then residually connected to the input of the temporal adaptive enhancement module. The output after residual connection is used as the output of the action adaptive enhancement module.
6. The continuous sign language recognition method based on action and temporal adaptive enhancement according to claim 5, characterized in that, Each of the first dilated convolutional block to the fourth dilated convolutional block includes a connected SE module and a 3D convolutional layer. The dilation rates of the 3D convolutional layers in the first dilated convolutional block to the fourth dilated convolutional block are different.
7. The continuous sign language recognition method based on action and temporal adaptive enhancement according to claim 5, characterized in that, In the first dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (1,1,1); in the second dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (2,1,1); in the third dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (3,1,1); in the fourth dilated convolution block, the kernel size of its 3D convolutional layer is 3×3×3, and the dilation rate is (4,1,1).
8. A continuous sign language recognition system based on action and temporal adaptive enhancement, characterized in that, include: An image preprocessing unit is used to preprocess the images in each image sequence to obtain a preprocessed image sequence. Training set construction unit, used to construct a training set based on the preprocessed image sequence; The continuous sign language recognition unit is used to store a neural network model for continuous sign language recognition and to use the neural network model to perform continuous sign language recognition on the input image sequence, thereby outputting the corresponding continuous sign language recognition result; The training unit is used to train the neural network model using the training set.
9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the continuous sign language recognition method based on action and temporal adaptive enhancement.
10. A computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the continuous sign language recognition method based on action and timing adaptive enhancement.