Dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM

Abstract

The invention provides a dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM, and aims to solve the problems that in the prior art, due to the fact that a large amount of invalid and repeated data exists in preprocessing of sign language video data, the network training amount is large, and the dynamic gesture time sequence feature extraction effect is affected. The method mainly comprises the steps of preprocessing, AFFA duplicate removal, training test set division, attitude feature extraction and fusion, recognition and classification, model parameter adjustment, recognition and detection and the like, a spatio-temporal feature extraction method is used, and the network feature extraction effect is improved through a 3D feature attention module; through the self-adaptive frame taking algorithm, invalid and repeated data are effectively screened out, the training overhead is reduced, the network training effect is improved, the proposed network model parameter quantity is smaller than that of a traditional model, and the training overhead is lower.

Description

Technical Field

[0001] The present invention relates to the technical field of sign language, and in particular to a dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM. Background Art

[0002] Sign language is the main means of communication for the hearing-impaired. In order to facilitate communication with non-hearing-impaired people, the development of sign language recognition technology is of great significance. Sign language involves the collaboration of multiple clues including gestures, facial expressions and body postures. However, the conversion of gestures makes it difficult for deep neural networks to automatically discover the implicit relationship between multiple visual clues. Some methods use external tools to solve this problem, such as using wearable devices for hand detection to handle shape changes and occlusions. Although there have been improvements, there are still problems that hinder the end-to-end learning of neural networks and repeated feature extraction.

[0003] Due to the continuous characteristics of dynamic sign language, more emphasis is placed on the use of temporal and spatial features. Dynamic gestures need to be temporally segmented. The common method is to decompose them into isolated word recognition problems. Inspired by the Long Short Term Memory (LSTM) network, many researchers use hierarchical attention networks to expand LSTM on the basis of considering structural information and feature attention mechanisms to complete dynamic sign language recognition tasks. The existing defect is that if the segmentation is inaccurate, semantic expression errors and other problems may occur in subsequent recognition. To solve this problem, some researchers use sensors to obtain gesture motion information to assist in completing dynamic sign language recognition tasks, but this method requires the subjects to wear motion sensor equipment, which causes certain troubles to sign language recognition work.

[0004] Secondly, the existing technology also has deficiencies in the preprocessing of sign language video data - specifically, there is a large amount of invalid and duplicate data in the data set, which not only makes the network training volume huge but also affects the extraction effect of dynamic gesture temporal features. In order to improve the utilization rate of temporal features, some methods use multiple sensors, such as visible light RGB cameras, depth cameras, millimeter wave radars, or calculate additional channels, such as optical flow, to improve their performance, resulting in huge model parameters and high network training volume, and limited improvement in application in actual scenarios. Gestures are temporally correlated and continuous in space. This dependence on space and time shows that the use of spatial and temporal features of gestures is particularly important, and existing methods are insufficient in this regard.

[0005] Therefore, how to propose a dynamic recognition method to make full use of the time-related information between frames and the hand position, shape, and direction encoding information in each frame is particularly important for improving the utilization rate of gesture features and improving the model recognition effect. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned prior art, the present application provides a dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM. The present invention uses a spatiotemporal feature extraction method and improves the network feature extraction effect through a 3D feature attention module. At the same time, the present invention effectively screens out invalid and duplicate data through an adaptive frame acquisition algorithm, reduces training overhead, and improves network training effect. The proposed network model has fewer parameters than the traditional model and has low training overhead.

[0007] The technical solution adopted by the present invention is as follows: A dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM comprises the following steps: S1: Sign language video recording, obtain dynamic sign language posture information through the camera, and remove the head and tail frames of the video; S2: The pre-processed video is processed by adaptive frame extraction through the AFFA algorithm to filter out invalid and duplicate data in the data set; S3: The preprocessed data set is fed into a 3D convolutional neural network for spatiotemporal feature extraction; S4: Integrate the extracted spatiotemporal features into 3D extrusion to stimulate the module for feature refinement, propose invalid features, and make useful features such as gesture posture more concentrated. Connect the output features with the fully connected layer, and finally output N classification probabilities through softmax (N is the number of gesture categories; after training, save the model and test it, perform N cross-validations on the public dataset (N is the number of dataset targets), verify the generalization of the model and save the best training model.

[0008] Furthermore, in the step S1, the head and tail frame removal operation is performed on the acquired sign language dynamic video to remove the hand gesture action segments of the start and stop actions in a single sign language video, and the total number of gesture action frames is fixed.

[0009] Furthermore, in step S2, the adaptive video framing method includes three steps of image sampling, image similarity comparison and image preservation in sequence. First, the video data is converted into image frame data, and the image similarity between two adjacent frames is calculated during the framing process, and the similarity between images is calculated using PC and GM through FSIM.

[0010] Furthermore, the calculation of FSIM includes the following steps: definition , It is an odd-symmetric, even-right orthogonal filter at scale n. For position x, scale n is to apply the above filter to obtain the vector: Its amplitude is: make: Then the 1D signal The PC is calculated as: in: The gradient feature GM can be calculated by the operator to calculate the gradients in the horizontal and vertical directions and calculate the final gradient: For two images, PC1, GM1, PC2, and GM2 can be calculated using the above formula. First, the similarity of PC is calculated as follows: GM similarity is calculated as follows: The similarity between PC and GM fusion is: ; Finally, the image FSIM is calculated as follows: .

[0011] Furthermore, after calculating the similarity FSIM of adjacent images, the discrimination threshold FSIM is initialized to 0.995 and the image segmentation number NUMmin is set to 16, and the video is framed according to the discrimination method.

[0012] Furthermore, the judgment method is: if FSIMy≥FSIMx, the previous frame image is saved, the video frame number NUM is updated, and the next frame of the current frame is taken in sequence, the image similarity is calculated and the FSIMx value is updated; if FSIMy<FSIMx, the previous frame image is discarded, and the next frame of the current frame is taken in sequence, the image similarity is calculated and the FSIMx value is updated.

[0013] Furthermore, in step S3, the spatiotemporal features of dynamic gestures are extracted through the 3D FACNN network module, and the utilization rate of the spatiotemporal features of dynamic gestures is enhanced through the 3D-SE attention module to extract feature information containing sign language context information.

[0014] Furthermore, after removing the output layer, the 3D FACNN network model structure includes a first convolutional layer, and its convolution size is , the convolution kernel is represented by four-dimensional data, which are time, width, length and channel information. The second and third layers of the network use the same hyperparameter settings, and the convolution kernel size is changed to , the number of feature layers are 64 and 128 respectively, the number of feature layers of the fourth and fifth layers is 128, and the sixth block consists of two fully connected layers.

[0015] Furthermore, in step S4, in order to filter the contextual information at different positions in the input feature map to retain useful information and suppress useless information, the spatial and temporal features between consecutive frames are filtered by cascading the 3D-SE feature attention module, and the short-term spatiotemporal dependency information is deeply encoded and characterized.

[0016] Furthermore, based on the 3D-SE feature attention module, its channel is expanded and added Convolution compresses channel information, making it a feature attention module that can meet the 3D video convolution task.

[0017] The beneficial effects of the present invention compared with the prior art are as follows: 1) The present invention uses a spatiotemporal feature extraction method and improves the network feature extraction effect through a 3D feature attention module.

[0018] 2) The present invention uses an adaptive frame-taking algorithm to effectively filter out invalid and duplicate data, reduce training overhead, and improve network training effects.

[0019] 3) The network model proposed in the present invention has fewer parameters than the traditional model and lower training overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of sign language recognition in the present invention; Figure 2 This is a flow chart of the adaptive video frame taking method in the present invention; Figure 3 A schematic diagram of discarding frames in the present invention; Figure 4 A schematic diagram of saving frames in the present invention; Figure 5 This is a network architecture diagram of the 3D FACNN-LSTM in the present invention; Figure 6 is a structural diagram of the 3D-SE module in the present invention; Figure 7 This is the LSTM classification network structure diagram in the present invention Figure 8 Schematic diagram of the attention module position accuracy test in the present invention; Fig. 9It is a schematic diagram of the accuracy test of the fusion strategy in the present invention. DETAILED DESCRIPTION

[0021] The specific implementation of the present invention will be described below in conjunction with the accompanying drawings.

[0022] like Figures 1 to 9 As shown, the present invention proposes a dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM, aiming to solve the problem in the prior art that a large amount of invalid and repeated data exists in the preprocessing of sign language video data, resulting in a huge amount of network training and affecting the effect of dynamic gesture temporal feature extraction.

[0023] Specifically, the steps include: Step S1: Recording a sign language video, obtaining dynamic sign language gesture information through a camera, and removing the head and tail frames of the video.

[0024] Step S2: The pre-processed video is processed by adaptive frame acquisition using the AFFA algorithm to filter out invalid and duplicate data in the data set.

[0025] Step S3: Send the preprocessed data set to the 3D convolutional neural network for spatiotemporal feature extraction.

[0026] Step S4: Integrate the extracted spatiotemporal features into the 3D extrusion to stimulate the module to refine features, propose invalid features, and make useful features such as gestures more concentrated. Connect the output features to the fully connected layer, and finally output N classification probabilities (N is the number of gesture categories) through softmax. After training, save the model and test it, perform N cross-validations on the public dataset (N is the number of dataset targets), verify the generalization of the model, and save the best training model.

[0027] In one embodiment of the present invention, in step S1, the head and tail frame removal operation is performed on the acquired sign language dynamic video to remove the hand gesture action segments of the start and stop actions in a single sign language video, and the total number of gesture action frames is fixed.

[0028] In one embodiment of the present invention, in step S2, the adaptive video frame method is as follows: Figure 2 As shown in the figure, it mainly consists of three parts: image sampling, image similarity comparison and image preservation. Firstly, the video data is converted into image frame data, and the image similarity between two adjacent frames is calculated during the frame sampling process. The similarity between images is calculated using PC and GM through FSIM.

[0029] Assumptions , It is an odd-symmetric, even-right orthogonal filter at scale n. For position x, scale n is to apply the above filter to obtain the vector: (1) Its amplitude is: (2) make: (3) (4) Then the 1D signal The PC is calculated as: (5) in: (6) The gradient feature GM can be calculated by the operator.

[0030] Calculate the horizontal and vertical gradients and calculate the final gradient: (7) For two images, PC1, GM1, PC2, and GM2 can be calculated using the above formula. First, the similarity of PC is calculated as follows: (8) GM similarity is calculated as follows: (9) The similarity between PC and GM fusion is: (10) Finally, the image FSIM is calculated as follows: (11) Read adjacent image frames in order according to the input video and calculate the similarity FSIMx of adjacent images according to formula (11). Initialize the discrimination threshold FSIMy to 0.995 and set the image segmentation number NUMmin to 16. Perform the frame operation on the video according to the following discrimination rules: ① If FSIMy ≥ FSIMx, save the previous frame image, update the video frame number NUM, and take the next frame of the current frame in sequence, calculate the image similarity and update the FSIMx value.

[0031] ② If FSIMy < FSIMx, discard the previous frame image, and take the next frame of the current frame in sequence, calculate the image similarity and update the FSIMx value.

[0032] like Figure 3 As shown in the figure, the adjacent image frames of the sign language video are taken in time order, and the FSIMy discrimination threshold is 0.995. Figure 3The image similarity FSIMx of the two images is 0.999, which is greater than FSIMy, indicating that the image similarity is too high, so the previous frame image is discarded. Figure 4 In the example, FSIMx is 0.931, which is less than FSIMy. Save the previous frame. Repeat steps ① and ② until the current video frame is taken. Compare NUM with the minimum frame value NUMmin: ① Initialize the discrimination threshold FSIMy to 0.995.

[0033] ②If NUM≥NUMmin, save the image to the training data set.

[0034] ③ If NUM < NUMmin, delete the locally saved image and increase the FSIMy value (by 0.001 each time), and re-perform the frame-taking operation on the current video. This step can significantly filter out invalid and duplicate data in the data set, reducing the total amount of data after the video frame is taken.

[0035] In one embodiment of the present invention, in step S3, dynamic sign language contains rich spatiotemporal features. In order to fully extract the temporal and spatial information of dynamic sign language, a 3D FACNN network module is designed for dynamic gesture spatiotemporal feature extraction. The architecture of the model is as follows: Figure 5 This part uses the 3D-SE attention module to enhance the network's utilization of dynamic gesture spatiotemporal features and extract feature information containing sign language context information.

[0036] First, the video data is preprocessed by adaptive frame acquisition method, and the data is normalized to At the same time, the length of the video unit of 16 frames is used as the length of the dynamic sign language video subsequence for data reading. The dynamic gesture spatial features are extracted through the 3DFACNN network and connected to the LSTM network to extract the temporal features for classification and recognition. After the output layer of the 3DFACNN network is removed, the structure consists of five functional blocks. The convolution size of the first convolution layer is The convolution kernel is represented by four-dimensional data, namely time, width, length and channel information, which are used to extract the time and space characteristics of the image. In order to extract the complete dynamic gesture features, the time dimension is initially set to 16, which means that 16 consecutive frames are input, that is, a complete dynamic sign language.

[0037] The second and third layers of the network use the same hyperparameter settings and change the convolution kernel size to (The size of the first convolution kernel is 5 5 5), the number of feature layers is 64 and 128 respectively, and the number of feature layers of the fourth and fifth layers is 128. The sixth block consists of two fully connected layers. The output of each block in the first five blocks is transferred to the lower layer through the maximum pooling layer for feature convolution operation processing. Except for the first layer, the size of the remaining pooling layers is , the step length is , the maximum pooling size of the first layer is , the step length is , setting a smaller size is to preserve the time information of the early feature map. In addition, in order to avoid the problem of overfitting causing the model to have a decline in test performance, it is necessary to use the regularization method Dropout after each functional layer of the network. This method randomly sets neurons to 0 during model training, making the network less sensitive to small perturbations of the input data, reducing the model's over-reliance on specific neurons, thereby reducing the risk of overfitting and improving the generalization ability of the model. In addition, a nonlinear activation unit Relu is used between each layer for activation, as shown in formula (12). It returns the input value when the network input is greater than zero, and returns zero when the input is less than zero. Using this function can not only avoid the gradient explosion problem, but also improve the network calculation effect and speed up the network fitting process.

[0038] (12) In one embodiment of the present invention, in step S4, in order to filter the context information at different positions in the input feature map to retain useful information and suppress useless information, the spatial and temporal features between consecutive frames are filtered by cascading the 3D-SE feature attention module, and the short-term spatiotemporal dependency information is deeply encoded and characterized. The 3D-SE feature attention module is integrated into the network for fine-grained gesture feature extraction. The purpose of this module is to improve the quality of network feature representation by explicitly modeling the interdependence between its convolutional feature channels. This mechanism allows the network to perform feature recalibration and learn to use global information to selectively emphasize information-rich features and suppress less useful features to enhance the network's representation and generalization capabilities. In addition, on this basis, its channels are expanded and added. Convolution compresses channel information, making it a feature attention module that can meet the 3D video convolution task. Figure 6 The feature attention module 3D-SE proposed by the present invention is shown.

[0039] 3D-SE is a computational unit that can be built on the input Mapping to feature maps Transformation In the horizontal connection between convolution blocks, as the convolution operator, and use represents the set of learned filter kernels, where represents the parameters of the cth filter. The output can then be written as ,in (13) represents convolution, , , . is a single value that represents the corresponding channel acting on X 2D spatial kernel of the channel. Since the output is calculated by summing over all channels, the channel correlation is embedded in In this step, the redundant information in the spatiotemporal features is filtered out to retain the effective feature information. The spatiotemporal features are downsampled by the fifth layer of the network through maximum pooling and sent to the gesture classification model LSTM for classification and recognition. The temporal and spatial gesture shape features output by 3D FACNN are processed into Sequence features, where T is the predetermined number of frames (the default is 16) and R is the feature output of each frame, so the input of the LSTM network is LSTM changes the traditional recurrent neural network that can only mine short-term dynamic information and cannot discover the relationship between long-term inputs. Figure 7 The internal structure of the LSTM classification network is shown. The network contains an input gate, an output gate, and a forget gate, which jointly refine the input features and are connected to the classifier to complete the classification and recognition task.

[0040] Different from the traditional recurrent neural network RNN, the LSTM network contains a loop feedback system for the input feature sequence , Is 3D FACNN in The feature vector extracted at each moment encodes the context information of the given time series to obtain the hidden state of the recurrent layer and output : (14) (15) Among them, Wih, Whh, Wh are the weight matrices connecting the input layer, hidden layer and output layer, b is the bias, is the hidden layer activation function. Hidden state According to the time series transmission, output It is input into the Softmax layer to make probability predictions for each dynamic sign language gesture.

[0041] After the model training is completed, the model is loaded and sign language recognition is performed. First, the camera acquires the manual dynamic video and performs video framing according to the adaptive framing algorithm proposed in the present invention. Then, the preprocessed data is sent to the spatiotemporal feature fusion network for prediction, and the prediction results are output and recorded to verify the model's sign language recognition performance.

[0042] Comparative experiment: To verify the effectiveness of the present invention, the following comparative experiment was conducted: The information of the verification data set of the present invention is as follows (1) ASL64: 10 volunteers wore red and green gloves on their left and right hands respectively and performed 64 sign language gestures in front of a camera. Each gesture was recorded five times, and a total of 3,200 sign language videos were recorded. The gestures in the videos were all from daily life.

[0043] (2) GMU-ASL: This dataset consists of 12 subjects performing 51 word-level sign language movements. The background of each sign language recording is different. The volunteers stood in fixed positions in several scenes and made different gestures according to the prompts. A total of 2,448 sign language videos were recorded.

[0044] (3) MA-ASL56: A self-built dataset of ten people, which records videos of 56 sign language gestures recorded by ten volunteers (aged 12 to 55 years old). This dataset contains a total of 2,800 video data.

[0045] The initial learning rate used in the present invention is , the weight decays to , momentum factor 0.9, hyperparameters Experience setting , the optimizer is SGD, and the training time series length is 16. The learning rate is reduced by half every 10 epochs. The training mini-batch is 6, and the total number of training rounds is 151. All experiments are implemented using the PyTorch library, and the model is trained using the NVIDIA GeForceGTX 2060 GPU graphics card. The 3DFACNN-LSTM dynamic sign language recognition network proposed in this invention is evaluated on three dynamic sign language video databases.

[0046] The amount of dynamic sign language video data after frame extraction is relatively large, and there are a large number of invalid and repeated data in the frame extraction data set, which makes the model training amount large, causing a certain impact on the extraction of dynamic gesture temporal features and thus affecting the network recognition accuracy. In order to compress the amount of training data and reduce the training time of the network model, and at the same time avoid the impact of invalid and repeated data on model recognition, the present invention proposes an adaptive video frame extraction method by calculating the similarity of adjacent image frames, and removing adjacent image frames with higher similarity through a discrimination method while retaining images with larger differences. The image similarity judgment threshold is adjusted according to the model training effect. This method does not require the segmentation operation of the front and back segments of the video data, simplifies the image preprocessing process, and reduces the time complexity. In addition, the screening of invalid and repeated data effectively reduces the training time of the network model and reduces the impact of invalid data on the network recognition efficiency. By calculating the FSIM image similarity of adjacent frames, the images that meet the conditions are eliminated, and the changes in the total amount of data in the data set, the model training time and the model recognition accuracy are recorded at the same time. The results are shown in Table 1.

[0047] Where NUMr is the total number of original images in the dataset. For each video, frames are taken at intervals of 4 frames and the head and tail frames are removed, totaling 5 frames. NUMs is the total number of images extracted using different thresholds. FSIM is the image similarity threshold. If it is higher than the set value, the previous frame is discarded; if it is lower than the set value, it is retained. When the video does not meet the minimum frame number requirement, the threshold is increased to increase the number of frames taken. T is the model training time (unit: hour), which is calculated by formula (16).

[0048] (16) in represents the model training time, represents the model validation and testing time, Indicates the number of network iterations (default 150).

[0049] Table 1 Adaptive frame acquisition parameter optimization FSIM (%) NUMr NUMs Accuracy(%) T (h) - 112000 - 95.56 11.375 0.999 112000 100832 95.59（+0.03） 10.625（-0.75） 0.998 112000 98651 95.68（+0.12） 9.542（-1.833） 0.995 112000 86779 95.12（-0.44） 9.292（-2.083） As shown in Table 1, when the threshold is controlled at 0.998, the original video frame count is reduced from 112,000 to 98,651, and the total number of frames is reduced by 11.9%, but the accuracy is slightly improved by 0.12%. By analyzing the data set screened after the AFFA improvement, it is found that compared with the previous data set, the number of duplicate data with high similarity has decreased. At the same time, the network model training time has been significantly improved compared with the previous data set, which has been reduced by 1.833 hours and 16.1%. Compared with the method without this strategy, the model training time and total training parameters have been greatly reduced, and the recognition accuracy has been slightly improved (+0.12%), which is due to the screening of duplicate and invalid data, proving the effectiveness of this method in improving model training time and reducing the amount of training data.

[0050] In order to explore the influence of the attention module 3D-SE proposed in this invention on the network recognition accuracy at different positions of the 3D convolutional neural network, comparative experiments were carried out. The experimental results are the experimental results after averaging n (n represents the number of targets in the data set) cross experiments. The experimental results are shown in Table 3.

[0051] In the table, FLOPs represents the number of floating point operations, Param represents the model training parameters, Baseline is the 3D convolutional neural network, Conv BlK is the bottleneck layer, and Bottle.1~3 are one to three functional layers. The 3D-SE attention module is inserted after the network bottleneck layer and different functional layers and the network test results are recorded. The fusion performance is shown in Table 3. Figure 8 The convergence performance test images of the model with 3D-SE attention module inserted after different layers are shown.

[0052] Table 2 Comparative experiment of 3D-SE fusion network Attention location Param(M) FLOPs Accuracy(%) Baseline 89.58 35.7 93.12 Conv BlK 89.58 35.7 92.16（-0.96） Bottle.1 89.58 35.7 93.34（+0.22） Bottle.2 89.58 35.7 94.67（+1.55） Bottle.3 89.58 35.7 92.95（-0.17） As can be seen from Table 2, compared with not using the 3D-SE feature module, when 3D-SE is placed after the three-layer functional block, the network test accuracy is improved the most (+1.55%). At the same time, the parameters provided by 3D-SE are negligible (the number of parameters is less than 0.01M), and the network training overhead does not increase significantly, which proves the effectiveness of 3D-SE. Figure 8 It can be seen that for dynamic sign language recognition, 3D CNN shows good convergence, and the network basically converges when the network iteration reaches 25 epochs.

[0053] In order to make full use of the temporal characteristics of dynamic gestures, further improve the network recognition accuracy, and improve the utilization rate of network features, the network is combined with the LSTM long-term and short-term network to explore the best combination. The temporal and spatial gesture shape features output by 3D FACNN are combined into a The sequence features are sent to the LSTM network for feature screening and classification. Table 3 shows several different combination strategies and their experimental results.

[0054] In Table 3, the Baseline is a 3D CNN network. After the network is fused with 3D-SE and LSTM, the network recognition efficiency is improved, with the highest improvement of 1.85%. This is because 3D-SE is committed to improving the utilization rate of continuous features in gesture space, and LSTM aims to strengthen useful features and suppress useless features in time and space dimensions. Although there are differences, they both improve the utilization performance of fine-grained features of network gestures to varying degrees, indicating the strengthening effect of this method in the channel features of dynamic gesture feature maps. After the network is fused with 3D-SE and LSTM long short-term memory network, the network recognition accuracy is further improved (+2.39%), indicating the effectiveness of the network fusion of 3D-SE and LSTM, and also showing the performance of 3D-SE in improving feature utilization. Fig. 9 The network convergence process under the three fusion modes is shown. The network becomes stable after 35 epochs, and the test accuracy does not fluctuate significantly, indicating the stability of this fusion method.

[0055] Table 3 Fusion strategy comparison experiment Method Param(M) FLOPs (G) Accuracy(%) Baseline 89.58 35.7 93.12 +3D-SE 89.58 35.7 94.82（+1.70） +LSTM 94.65 36.2 94.97（+1.85） +3D-SE-LSTM 94.65 36.2 95.51（+2.39） The present invention conducted comparative experiments on a public dynamic sign language dataset and a self-built dynamic sign language dataset of ten people, and compared the dynamic sign language recognition method based on adaptive video frame extraction method and 3D FACNN-LSTM proposed in the present invention with the existing method. Table 4 lists the average test accuracy of multiple subjects obtained in the public dataset using the leave-one-out cross-validation evaluation, and the specific test results are shown in Table 4. Among them, ASL64 and GMU-ASL51 respectively represent the test accuracy on the public dataset.

[0056] Table 4 Experimental comparison with existing methods on public datasets Method Param(M) FLOPs (G) ASL64 (%) GMU-ASL (%) M-SSA 63.10 31.7 94.75 92.72 Fine:hand 102 43.0 95.12 93.00 T-singer 92.7 36.9 97.43 95.32 3D FACNN-LSTM 94.65 36.2 99.05 96.75 As can be seen from the table, the dynamic sign language recognition network (3D FACNN-LSTM) proposed in the present invention has greatly improved the recognition accuracy compared with the existing methods. All methods performed best on ASL64. This is because the ASL64 public data set highlights the gesture features by wearing gloves during recording, which improves the distinction between dynamic gestures and background. The method proposed in the present invention achieved the highest recognition rate (99.05%) on this data set, and there was no significant increase in model training parameters and time complexity. This is due to the role of the adaptive frame method in eliminating invalid data and duplicate data in the data set, which reduces the amount of network model training.

[0057] Table 5 Experimental results comparing with existing methods on the self-built dataset MA-ASL56 Method Param(M) Flops (G) 3-people (%) 5-people (%) 10-people (%) M-SSA 63.10 31.7 95.61 94.71（-0.90） 92.49（-3.12） Fine:hand 102 43.0 96.90 95.97（-0.93） 93.88（-3.02） T-singer 92.7 36.9 97.32 96.83（-0.60） 94.53（-2.79） 3D FACNN-LSTM 94.65 36.2 98.43 97.86（-0.57） 95.56（-2.87） At the same time, a comparative experiment was conducted on a self-built dynamic sign language data set of ten people. The experimental results are shown in Table 5, which shows the accuracy comparison between the method proposed in the present invention and other methods in the recognition and detection of three, five, and ten-person data sets. As the number of targets increases, there are more features and change patterns in the data set, requiring the network to have more complex feature representation capabilities. Therefore, when the number of people detected increases, the detection accuracy decreases to varying degrees. Despite this, the method proposed in the present invention achieves the best recognition effect compared with other methods, proving that the proposed temporal feature fusion network has good processing capabilities and feature learning capabilities for complex data.

[0058] From the experimental results in Tables 4 and 5, it can be seen that the method proposed in the present invention has achieved the best performance results in terms of accuracy, indicating the advantages of integrating the 3D-SE feature attention module and the long short-term memory network in improving the utilization rate of fine-grained features of dynamic gestures, and proving the effectiveness of the method of the present invention. In addition, the experimental results on three different data sets show that the method of the present invention has good generalization ability.

[0059] The above description is an explanation of the present invention, not a limitation of the present invention. The scope of the present invention is defined in the claims. Any form of modification may be made within the scope of protection of the present invention.

Claims

1. A dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM, characterized by: The steps include: S1: Sign language video recording, obtain dynamic sign language posture information through the camera, and remove the head and tail frames of the video; S2: The pre-processed video is processed by adaptive frame taking through the AFFA algorithm to filter out invalid and duplicate data in the data set; S3: The preprocessed data set is fed into a 3D convolutional neural network for spatiotemporal feature extraction; S4: Integrate the extracted spatiotemporal features into 3D extrusion to stimulate the module to refine features and propose invalid features to make useful features such as gesture posture more concentrated, and connect the output features with the fully connected layer, and finally output N classification probabilities through softmax (N is the number of gesture categories; After training, save the model and test it, perform N cross-validations on the public dataset (N is the number of dataset targets) to verify the generalization of the model and save the best training model.

2. The dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM as claimed in claim 1, characterized in that: In the step S1, the head and tail frame removal operation is performed on the acquired sign language dynamic video to remove the hand gesture action segments of the start and stop actions in a single sign language video, and the total number of gesture action frames is fixed.

3. The dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM as claimed in claim 1, characterized in that: In step S2, the adaptive video framing method includes three steps of image sampling, image similarity comparison and image preservation. First, the video data is converted into image frame data, and the image similarity between two adjacent frames is calculated during the framing process, and the similarity between images is calculated using PC and GM through FSIM.

4. The dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM as claimed in claim 3, characterized in that: The calculation of FSIM includes the following steps: definition , It is an odd-symmetric, even-right orthogonal filter at scale n. For position x, scale n is to apply the above filter to obtain the vector: Its amplitude is: make: Then the 1D signal The PC is calculated as: in: The gradient feature GM can be calculated by the operator to calculate the gradients in the horizontal and vertical directions and calculate the final gradient: For two images, PC1, GM1, PC2, and GM2 can be calculated using the above formula. First, the similarity of PC is calculated as follows: GM similarity is calculated as follows: The similarity between PC and GM fusion is: ； Finally, the image FSIM is calculated as follows: 。 5. The dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM as claimed in claim 4, characterized in that: After calculating the similarity FSIM of adjacent images, the discrimination threshold FSIM is initialized to 0.995 and the number of image segmentation NUMmin is set to 16, and the video is framed according to the discrimination method.

6. The method for dynamic sign language recognition based on AFFA and 3D FACNN-LSTM as claimed in claim 5, characterized in that: The determination method is: if FSIMy≥FSIMx, the previous frame image is saved, the video frame number NUM is updated, and the next frame of the current frame is sequentially taken, the image similarity is calculated and the FSIMx value is updated; if FSIMy<FSIMx, the previous frame image is discarded, and the next frame of the current frame is sequentially taken, the image similarity is calculated and the FSIMx value is updated.

7. The dynamic sign language recognition method based on AFFA and 3D FACNN-LSTM as claimed in claim 1, characterized in that: In step S3, the spatiotemporal features of dynamic gestures are extracted through the 3D FACNN network module, and the utilization of the spatiotemporal features of dynamic gestures by the 3D-SE attention module is enhanced to extract feature information containing sign language context information.

8. The method for dynamic sign language recognition based on AFFA and 3D FACNN-LSTM as claimed in claim 7, characterized in that: After removing the output layer, the 3D FACNN network model structure includes the first convolution layer, whose convolution size is , the convolution kernel is represented by four-dimensional data, which are time, width, length and channel information. The second and third layers of the network use the same hyperparameter settings, and the convolution kernel size is changed to , the number of feature layers are 64 and 128 respectively, the number of feature layers of the fourth and fifth layers is 128, and the sixth block consists of two fully connected layers.

9. The method for dynamic sign language recognition based on AFFA and 3D FACNN-LSTM as claimed in claim 1, characterized in that: In step S4, in order to filter the contextual information at different positions in the input feature map to retain useful information and suppress useless information, the spatial and temporal features between consecutive frames are filtered by cascading the 3D-SE feature attention module, and the short-term spatiotemporal dependency information is deeply encoded and characterized.

10. The method for dynamic sign language recognition based on AFFA and 3D FACNN-LSTM as claimed in claim 9, characterized in that: Based on the 3D-SE feature attention module, its channel is expanded and added Convolution compresses channel information, making it a feature attention module that can meet the 3D video convolution task.

Images