Method and system for sign language recognition based on skeleton joint and image modal fusion

Abstract

The application belongs to the technical field of sign language recognition, and discloses a sign language recognition method and system based on skeleton joint point and image modal fusion, which comprises the following steps: dividing sign language skeleton joint point data into regions, inputting the data into a preset graph convolutional neural network to obtain skeleton joint point feature vectors of each region; inputting sign language RGB image data into a ResNet-18 network model based on a multi-dimensional attention module to obtain image feature vectors; inputting the skeleton joint point feature vectors of each region and the image feature vectors into Bi-LSTM models respectively to obtain sequence features of sign language skeleton joint point data and sequence features of sign language RGB image data; performing sign language prediction based on the respective sequence features, and finally performing late fusion to obtain a final sign language recognition result. The application improves the accuracy and robustness of the sign language recognition result and enhances the application of the sign language recognition system in actual environments.

Description

Technical Field

[0001] This invention belongs to the field of sign language recognition technology, specifically relating to a sign language recognition method and system based on skeletal joint point and image modality fusion. Background Technology

[0002] Sign language, as one of the main means of communication for hearing-impaired individuals, is an important medium for communication between hearing-impaired individuals and between hearing-impaired individuals and the general public. Since the general public generally cannot understand or use sign language, sign language recognition systems play a crucial role in daily communication with hearing-impaired individuals, possessing immense application value and a wide range of application scenarios.

[0003] With the rapid development of image technology and artificial intelligence, current research on sign language recognition focuses on processing single-modal feature data using deep learning models. This includes using deep learning models to extract and process features from RGB image data, human skeletal joint data, or depth image data to obtain recognition results. While this approach has achieved considerable success in recent years, single-modal sign language recognition schemes often neglect visual or semantic information, leading to incomplete expression. Effective fusion of multimodal data can compensate for the shortcomings of single-modal data and mitigate the impact of complex environments in practical sign language recognition system applications, thus achieving greater robustness. In multimodal sign language recognition tasks, effective feature extraction from various modalities and effective fusion of multimodal data are currently key research areas.

[0004] RGB image data provides rich visual information, but it is easily affected by environmental factors such as lighting and complex backgrounds. Skeletal joint data, on the other hand, is less affected by environmental interference and provides more accurate pose information; these two modalities are complementary. However, RGB image data contains a large amount of redundant information, which can reduce the accuracy and robustness of recognition. For skeletal joint data, previous studies have typically used the detected joint positions directly as feature data for sign language recognition. This method is easily affected by pose changes and occlusion, and cannot capture the movement and spatial relationships between hands. Regarding modal fusion methods, including early fusion, hybrid fusion, and late fusion, current research shows that early fusion is more effective for fusing information from the same modality, while late fusion is more effective for fusing information from different modalities. In conclusion, how to effectively extract features from RGB image data and skeletal joint data, and how to perform effective modal fusion, are pressing problems that need to be solved in current sign language recognition research. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a sign language recognition method and system based on skeletal joint point and image modality fusion, aiming to improve the accuracy and robustness of sign language recognition systems in practical application scenarios.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] This invention is a sign language recognition method based on skeletal joint point and image modality fusion, comprising the following steps:

[0008] (1) Divide the acquired sign language skeletal joint data into regions and input them into a pre-defined graph convolutional neural network to obtain the feature vector V of each region. a V b and V c The specific steps are as follows:

[0009] (1a) Convert the acquired sign language video into an m-dimensional image sequence G = {g1, g2, g3, ..., g...} m For each frame image g i Using a skeletal joint detection model to detect skeletal joints, we can obtain results related to g. i The corresponding skeletal joint sequence P i ={keypoint i,1 keypoint i,2 ,…,keypoint i,n Each frame contains data for n skeletal joint points.

[0010] (1b) The obtained skeletal joint sequence P i Based on the human body structure, three regions S are obtained. a S b and S c These represent the three areas: left hand, right hand, and limbs.

[0011] (1c) For each frame image, the corresponding skeletal joint sequence P i The three partitions S a S b and S c Any partition S in j The skeletal joint data within the partition is processed using a graph structure. Assuming there are J skeletal joints in the partition, each joint is treated as a node, denoted by q. The two-dimensional coordinates of each joint are used as its feature information. An incomplete undirected graph is constructed based on the natural connectivity of human skeletal joints, yielding the corresponding adjacency matrix A and node feature matrix Q, where A∈R. J×J , Q∈R J×2 .

[0012] (1d) Take each S in the m-dimensional image j The adjacency matrix A and the node feature matrix Q corresponding to each partition are input into a pre-defined graph convolutional neural network model for feature extraction, yielding the corresponding feature vector V. j ,in:

[0013]

[0014] In the formula: v j S represents the value of each frame of the image. j The feature vector obtained from partitioning, d is v j The dimension of the feature vector.

[0015] For each frame of image S j The specific steps for inputting the adjacency matrix A and node feature matrix Q of the partition into the graph convolutional neural network for processing are as follows:

[0016] (1d1) In graph convolutional neural networks, the representation of a node is updated by aggregating information from neighboring nodes. Suppose we want to update node q. i The updated formula is:

[0017]

[0018] Where q k q i The adjacent nodes, θ(q) i ) represents q i The set of adjacent nodes is calculated from the adjacency matrix A, ||θ(q) i )|| represents the number of neighboring nodes, W is the parameter to be learned in the graph convolutional layer; σ represents the node feature update function of the graph convolutional layer, q i S represents j Any node in the partition.

[0019] (1d2) In the graph convolutional neural network model, two graph convolutional layers are used to fuse global and local graph structure information to extract richer feature information. The ReLU function is used as the activation function, and the specific expression is as follows:

[0020] Q' = ReLU(GCN(AQW1))

[0021] Q” = ReLU(GCN(AQ'W2))

[0022] In the formula: Q′ is the node feature matrix obtained by the first graph convolutional layer, Q″ is the node feature matrix obtained by the second graph convolutional layer, and W1 and W2 are the parameters to be learned by the first and second graph convolutional layers, respectively.

[0023] (1d3) Connecting a fully connected layer yields the final feature vector v. The fully connected layer maps the feature map output by the graph convolutional layer into a d-dimensional feature vector, as shown in the following expression:

[0024] v = FC({Q”})

[0025] (2) Input the acquired sign language RGB image data into a ResNet-18 network model based on a multidimensional attention module to obtain the feature vector V. d The construction of the ResNet-18 model based on multi-dimensional attention modules involves sequentially embedding spatial attention modules and channel attention modules into each residual block of the ResNet-18 network model. This allows the network model to focus more on the motion information of the hand and arm, removing redundant information. The specific steps are as follows:

[0026] (2a) In each residual block, the feature map F after being processed by a 3*3 convolutional layer is input into the spatial attention module to obtain the spatial attention map M. S , where F∈R H×W×C H, W, and C represent the height, width, and number of channels of the feature map, respectively, as shown in the following expressions:

[0027] M S =σ(f S (GAP(F),GMP(F))

[0028] GAP(F) and GMP(F) represent global average pooling and global max pooling operations, respectively, used to extract important spatial regions from the feature map F. Then, a convolution operation f with a 7x7 kernel is applied. S The extracted spatial features are further convolved, and finally, an activation function is applied to obtain the spatial attention map M. S .

[0029] (2b) Spatial attention map M S The dot product of feature map F and feature map F' yields the processed feature map F', expressed as follows:

[0030]

[0031] (2c) Input the feature map F' into the channel attention module to obtain the channel attention map M. C The expression is as follows:

[0032] M C =σ(g C (GAP(F)+GMP(F))

[0033] GAP(F) and GMP(F) are used to adjust the importance of each channel, and then a multi-layer fully connected perceptron g is used. C The extracted channel features are concatenated and then passed through an activation function to obtain the channel attention map M. C .

[0034] (2d) Channel attention map M C The dot product of feature map F' and feature map F' yields the final feature map F', as shown in the following expression:

[0035]

[0036] (3) The feature vector V a V b and V c To conduct early integration, with V d The data are input into a Bi-LSTM model for processing to obtain the sequence features of the sign language skeletal joint data and the sequence features of the sign language RGB image data.

[0037] (4) The sequence features of the obtained sign language skeletal joint data and the sequence features of the sign language RGB image data are respectively input into the classification network to obtain the sign language prediction results of each modality. The two prediction results are then fused to obtain the final sign language recognition result. The specific steps are as follows:

[0038] (4a) The sequence features of the sign language skeletal joint data and the sequence features of the sign language RGB image data obtained after processing by the Bi-LSTM model are respectively input into the Softmax classifier for single-modal sign language prediction, and the prediction result Y is obtained. skeleton and Y RGB .

[0039] (4b) Regarding the predicted result Y skeleton and Y RGB Late-stage fusion is performed to obtain the final sign language recognition result Y, where the fusion expression is:

[0040] Y = Y skeleton +αY RGB

[0041] In the formula, α is the fusion parameter, which takes a value between 0 and 1.

[0042] On the other hand, the present invention provides a sign language recognition system based on skeletal joint point and image modality fusion, which mainly includes the following modules:

[0043] Sign language data acquisition module: Uses a monocular camera to acquire sign language video, converts it into an image sequence to obtain sign language RGB image data, and then uses a skeletal joint detection model to obtain sign language skeletal joint data.

[0044] Skeletal joint feature extraction module: The acquired sign language skeletal joint data is divided into regions and input into a preset graph convolutional neural network to obtain the skeletal joint feature vector of each region.

[0045] RGB Image Feature Extraction Module: This module inputs sign language RGB image data into a ResNet-18 network model based on a multi-dimensional attention module to obtain the feature vectors of the RGB images. The ResNet-18 model based on the multi-dimensional attention module is constructed by sequentially embedding spatial attention and channel attention modules into each residual block of the ResNet-18 network model. This allows the network model to focus more on hand and arm movement information and remove redundant information.

[0046] The temporal feature extraction module based on Bi-LSTM inputs the feature vectors obtained from the skeletal joint feature extraction module and the RGB image feature extraction module into the Bi-LSTM model to obtain the sequence features of their respective modalities.

[0047] Late fusion module: The sequence features of sign language skeletal joint data and the sequence features of sign language RGB image data are respectively input into the classification network to obtain the sign language prediction results of each modality. The two prediction results are then fused in the late stage to obtain the final sign language recognition result.

[0048] The beneficial effects of this invention are: This invention uses skeletal joint data and RGB image data of sign language as input, which is different from the traditional method of directly using the position of skeletal joints as feature input. It uses a partitioning strategy and graph convolutional neural network to process skeletal joint data, extracting the motion information and spatial information of the hand and arm contained in the skeletal joint data, and avoiding the influence of posture changes and limb occlusion.

[0049] This invention uses a ResNet-18 network model with embedded spatial attention and channel attention modules to extract visual and detail information of the hand and arm contained in RGB images and remove redundant information.

[0050] This invention inputs the feature vectors of two modalities into a Bi-LSTM network model to extract the temporal features of each modality. Finally, based on the sequence features of the obtained sign language skeletal joint data and the sequence features of the sign language RGB image data, sign language prediction is performed separately. The two prediction results are then fused to obtain the final sign language recognition result.

[0051] This invention improves the accuracy and robustness of sign language recognition results, and enhances the application of sign language recognition systems in real-world environments. Attached Figure Description

[0052] Figure 1This is a structural diagram of the model of the present invention.

[0053] Figure 2 This is a schematic diagram of the skeletal joint point region division of the present invention.

[0054] Figure 3 This is a structural block diagram of the present invention. Detailed Implementation

[0055] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0056] like Figure 1-3 As shown, this invention discloses a sign language recognition method based on skeletal joint point and image modality fusion. The sign language recognition method is implemented through a sign language recognition system, which includes:

[0057] Sign language data acquisition module: Use a PC-based monocular camera to acquire sign language videos, convert them into image sequences to obtain sign language RGB image data, and then use the MediaPipe skeletal joint detection model to obtain sign language skeletal joint data;

[0058] Skeletal joint feature extraction module: The acquired sign language skeletal joint data is divided into regions and input into a preset graph convolutional neural network to obtain the skeletal joint feature vector of each region.

[0059] RGB image feature extraction module: Input sign language RGB image data into the ResNet-18 network model based on multi-dimensional attention module to obtain the feature vector of RGB image. The ResNet-18 model based on multi-dimensional attention module is constructed by sequentially embedding spatial attention module and channel attention module in each residual block of the ResNet-18 network model, so that the network model can pay more attention to the motion information of hand and arm and remove redundant information.

[0060] The temporal feature extraction module based on Bi-LSTM: The feature vectors obtained from the skeletal joint feature extraction module and the RGB image feature extraction module are respectively input into the Bi-LSTM model to obtain the sequence features of their respective modalities;

[0061] Late fusion module: The sequence features of sign language skeletal joint data and the sequence features of sign language RGB image data are respectively input into the classification network to obtain the sign language prediction results of each modality. The two prediction results are then fused in the late stage to obtain the final sign language recognition result.

[0062] Based on this sign language recognition system, the sign language recognition method of the present invention includes the following steps:

[0063] Step 1: Divide the acquired sign language skeletal joint data into regions and input them into a pre-set graph convolutional neural network to obtain the feature vector V of each region. a V b and V c .

[0064] Specifically, this invention acquires sign language videos, fixes the resolution of all sign language videos at 640*480, and the frame rate at 30FPS. For each sign language video, it processes it according to the following steps:

[0065] Step 1-1: Process the acquired sign language video into a fixed 50-frame image sequence G = {g1, g2, g3, ..., g} using a keyframe extraction algorithm. 50 For each frame image g i Using the MediaPipe skeletal joint detection model for skeletal joint detection, we can obtain results similar to g. i The corresponding skeletal joint sequence P i ={keypoint i,1 keypoint i,2 ,…,keypoint i,51 The image contains 2D pixel coordinates of 51 skeletal joints, including 21 skeletal joints for the left hand, 21 skeletal joints for the right hand, and 9 skeletal joints for the limbs.

[0066] Step 1-2: Obtain the skeletal joint sequence P i Divide the area according to human anatomy, such as Figure 2 As shown, three partitions S are obtained. a S b and S c These represent the three areas: left hand, right hand, and limbs.

[0067] Steps 1-3: For each frame of image, the corresponding skeletal joint sequence P i The three partitions S a S b and S c Any partition S in j Perform graph structure processing on the skeletal joint data within the partition, assuming S j The partition contains J skeletal joints, which are treated as nodes, denoted by q. The two-dimensional coordinates of each skeletal joint are used as the node's feature information. An incomplete undirected graph is constructed based on the natural connectivity of human skeletal joints, yielding the corresponding adjacency matrix A and node feature matrix Q, where A∈R. J×J, Q∈R J×2 ;

[0068] Steps 1-4: Extract each S from the 50 frames corresponding to each sign language video. j The adjacency matrix A and node feature matrix Q corresponding to each partition are input into a pre-defined graph convolutional neural network model for feature extraction, yielding the feature vector V for each partition. j ,in:

[0069]

[0070] In the formula: v j S represents the value of each frame of the image. j The feature vector obtained from partitioning, v j The dimension is 512.

[0071] For each frame of image S j The specific steps for inputting the adjacency matrix A and node feature matrix Q of the partition into the graph convolutional neural network for processing are as follows:

[0072] Step 1-4-1: In a graph convolutional neural network, the representation of a node is updated by aggregating information from neighboring nodes. Assume the node q is being updated. i The updated formula is:

[0073]

[0074] Where q k q i The adjacent nodes, θ(q) i ) represents q i The set of adjacent nodes is calculated from the adjacency matrix A, ||θ(q) i )|| represents the number of neighboring nodes, W is the parameter to be learned in the graph convolutional layer; σ represents the node feature update function of the graph convolutional layer, q i S represents j Any node in the partition.

[0075] Step 1-4-2: In the graph convolutional neural network model, two graph convolutional layers are used to fuse global and local graph structure information, extracting richer feature information. The ReLU function is used as the activation function, and the specific expression is as follows:

[0076] Q' = ReLU(GCN(AQW1))

[0077] Q” = ReLU(GCN(AQ'W2))

[0078] In the formula: Q′ represents the node feature matrix obtained by the first graph convolutional layer, Q″ represents the node feature matrix obtained by the second graph convolutional layer, and W1 and W2 are the parameters to be learned by the first and second graph convolutional layers, respectively.

[0079] Step 1-4-3: Connect a fully connected layer to obtain the final feature vector v. The fully connected layer maps the feature map output by the graph convolutional layer into a d-dimensional feature vector, as shown in the following expression:

[0080] v = FC({Q”}).

[0081] Step 2: For the obtained RGB images, crop and convert them to images with a resolution of 224*224. Input each frame of the image into a ResNet-18 model based on a multi-dimensional attention module. This model is constructed by sequentially embedding spatial attention modules and channel attention modules into each residual block of the ResNet-18 network model, allowing the network model to pay more attention to the motion information of the hand and arm and remove redundant information. The specific steps are as follows:

[0082] Step 2-1: In each residual block, the feature map F processed by a 3*3 convolutional layer is input into the spatial attention module to obtain the spatial attention map M. S , where F∈R H×W×C H, W, and C represent the height, width, and number of channels of the feature map, respectively, as shown in the following expressions:

[0083] M S =σ(f S (GAP(F),GMP(F))

[0084] In the formula: GAP(F) and GMP(F) represent global average pooling and global max pooling operations, respectively, used to extract important spatial regions in the feature map F, and then a convolution operation f with a 7*7 kernel is applied. S The extracted spatial features are further convolved, and finally, an activation function is applied to obtain the spatial attention map M. S ;

[0085] Step 2-2: The spatial attention map M obtained in Step 2-1... S The dot product of feature map F and feature map F' yields the processed feature map F', expressed as follows:

[0086]

[0087] Step 2-3: Input the feature map F' obtained in step 2-2 into the channel attention module to obtain the channel attention map M. C The expression is as follows:

[0088] MC =σ(g C (GAP(F)+GMP(F))

[0089] Where: g C This represents a multilayer fully connected perceptron, where GAP(F) and GMP(F) are used to adjust the importance of each channel, and then the multilayer fully connected perceptron g is used. C The extracted channel features are concatenated and then passed through an activation function to obtain the channel attention map M. C ;

[0090] S2-4: Channel attention map M C The dot product of feature map F' and feature map F' yields the final feature map F', as shown in the following expression:

[0091]

[0092] Step 3: Convert the feature vectors V of each region obtained in Step 1 into... a V b and V c Perform early fusion, and combine it with the feature vector V obtained in step 2. d The data are input into a Bi-LSTM model for processing to obtain the sequence features of sign language skeletal joint data and sign language RGB image data.

[0093] Step 4: Based on the sequence features of the sign language skeletal joint data and the sequence features of the sign language RGB image data obtained in Step 3, input them into the classification network to obtain the sign language prediction results for each modality. Then, perform late fusion on these two prediction results to obtain the final sign language recognition result. The specific steps are as follows:

[0094] Step 4-1: Input the sequence features of the sign language skeletal joint data and the sequence features of the sign language RGB image data obtained after processing by the Bi-LSTM model into the Softmax classifier for single-modal sign language prediction, and obtain the prediction result Y. skeleton and Y RGB ;

[0095] Step 4-2, for the predicted result Y skeleton and Y RGB Late-stage fusion is performed to obtain the final sign language recognition result Y, where the fusion expression is:

[0096] Y = Y skeleton +αY RGB ,

[0097] In the formula, α is the fusion parameter, which takes a value between 0 and 1.

[0098] This invention comprehensively utilizes sign language skeletal joint data and RGB image data. It employs a pre-defined graph convolutional neural network model based on a partitioning strategy and a ResNet-18 network model based on a multi-dimensional attention module to effectively extract features from the data of each modality. Then, a Bi-LSTM model is used to extract the temporal features of each modality. Finally, the prediction results of the two modalities are fused in a late stage, which improves the accuracy and robustness of sign language recognition results and enhances the application of the sign language recognition system in real-world environments.

[0099] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for sign language recognition based on skeleton joint and image modality fusion, characterized in that: The sign language recognition method is implemented by a sign language recognition system, and specifically, the sign language recognition method comprises the following steps: Step 1, divide the obtained sign language skeleton joint data into regions, and input into a preset graph convolutional neural network respectively to obtain feature vectors V a , V b , and V c ; Step 2, input the obtained sign language RGB image data into a ResNet-18 network model based on a multi-dimensional attention module to obtain a feature vector V d ; Step 3, early fusion of the feature vectors V a , V b and V c obtained in step 1, and the feature vectors V d obtained in step 2 are respectively input into a Bi-LSTM model for processing to obtain sequence features of the sign language skeletal joint data and sequence features of the sign language RGB image data; Step 4, the sequence features of the sign language skeletal joint data obtained in step 3 and the sequence features of the sign language RGB image data are respectively input into a classification network to obtain sign language prediction results of each mode, and the two prediction results are late fused to obtain a final sign language recognition result; In the step 1, the feature vector V of each region is obtained a , b and V c Specifically includes the following steps: Step 1-1, the obtained sign language video is converted into an m-dimensional image sequence G = {g1, g2, g3, …, g i … m}, for each frame image g i , a MediaPipe skeleton joint detection model is used for skeleton joint detection to obtain a corresponding skeleton joint sequence P i = {keypoint i , keypoint i,1 , …, keypoint i,2} corresponding to each frame image g i,51 , containing 2D pixel coordinates of 51 skeleton joints, including 21 skeleton joints of the left hand, 21 skeleton joints of the right hand and 9 skeleton joints of the limbs; Step 1-2, the obtained skeleton joint sequence P i According to the human structure, three sub-regions S a , S b and S c are obtained, representing the left hand, the right hand and the limb respectively; Step 1-3, for each frame of image corresponding to the sequence of bone joints P i Any one of the three partitions S a , S b and S c in P j The skeleton joint data in the partition is processed by graph structure, assuming that the number of skeleton joints in the partition S j is J, the skeleton joint is taken as a node, the node is denoted by q, the two-dimensional coordinate information of each skeleton joint is taken as the characteristic information of the node, an incomplete undirected graph is established according to the natural connectivity of the human body skeleton joint, and the corresponding adjacency matrix A and node characteristic matrix Q are obtained, wherein A∈R J×J , Q∈R J×2 ; Step 1-4: each S in the m-dimensional image is divided into a plurality of sub-regions, and each sub-region is represented by a node j The adjacency matrix A and the node feature matrix Q corresponding to the partition are input as input data into a preset graph convolutional neural network model respectively for feature extraction, to obtain a corresponding feature vector V j wherein: ， In the formula: v j S represents the value of each frame of the image. j The feature vector obtained from partitioning, d is v j The dimension of the feature vector; specifically including the following steps: Step 1-4-1, in the graph convolutional neural network, the representation of a node is updated by aggregating the information of adjacent nodes, assuming that the updated node q i The update formula is: ， wherein: denotes q i adjacent nodes of the node, denotes q i a set of adjacent nodes, calculated by an adjacency matrix A, denotes the number of adjacent nodes, and W is a parameter to be learned by the graph convolution layer; denotes a node feature update function of the graph convolution layer, q i denotes S j any one node in the partition; Step 1-4-2, in the graph convolutional neural network model, two layers of graph convolutional layers are used to fuse global and local graph structure information and extract feature information, and a ReLU function is used as an activation function, and the specific expression is as follows: ， In the formula: is the node feature matrix processed by the first layer of graph convolutional layer, is the node feature matrix processed by the second layer of graph convolutional layer, and W1 and W2 are parameters to be learned by the first layer of graph convolutional layer and the second layer of graph convolutional layer, respectively. Step 1-4-3, a full connection layer is connected to obtain a final feature vector v, and the full connection layer maps the feature map output by the graph convolutional layer into a d-dimensional feature vector, and the expression is as follows: ； The ResNet-18 network model based on the multi-dimensional attention module is constructed by sequentially embedding a spatial attention module and a channel attention module in each residual block in the ResNet-18 network model, and the specific construction process comprises the following steps: Step 2-1, in each residual block, the feature map F processed by the 3*3 convolution layer is input into the spatial attention module to obtain the spatial attention map M S , where F ∈ R H×W×C , H, W and C represent the height, width and channel number of the feature map respectively, and the expression is as follows: ， In the formula, GAP(F) and GMP(F) respectively represent a global average pooling operation and a global maximum pooling operation, which are used to extract important spatial regions in the feature map F, and then a convolution operation f with a convolution kernel of 7*7 is used to further convolve the extracted spatial features, and finally an activation function is used to obtain the spatial attention map M S S ;​ Step 2-2, obtaining a spatial attention map M from step 2-1 S The processed feature map F' is obtained by point multiplication of the feature map F, and the expression is as follows: ； Step 2-3, input the feature map F' obtained in step 2-2 into the channel attention module to obtain a channel attention map M C The expression is as follows: ， In the formula: g C represents a multi-layer fully connected layer perceptron, and is used to adjust the importance of each channel, and then a multi-layer fully connected layer perceptron g C is used to splice the extracted channel features, and finally an activation function is used to obtain a channel attention map M C ; S2-4: Channel attention map M is obtained by multiplying the feature map F’ with the channel attention map M C The final feature map F” is obtained by multiplying the feature map F’ with the channel attention map M, and the expression is as follows: 。 2. The sign language recognition method based on skeleton joint and image modal fusion according to claim 1, characterized in that: The step 4 specifically comprises the following steps: Step 4-1, for the sequence features of the sign language skeletal joint data obtained after processing by the Bi-LSTM model and the sequence features of the sign language RGB image data, respectively input to the Softmax classifier for single-modal sign language prediction to obtain the prediction result Y skeleton and Y RGB ; Step 4-2, for the predicted result Y skeleton and Y RGB perform late fusion to obtain a final sign language recognition result Y, wherein the fusion expression is: ， In the formula, α is a fusion parameter, and the value is between 0 and 1.

3. The sign language recognition method based on skeleton joint and image modal fusion according to claim 1, characterized in that: The sign language recognition system comprises: A sign language data acquisition module: a monocular camera is used to acquire sign language videos, and sign language RGB image data is obtained by converting the sign language videos into image sequences, and sign language skeletal joint data is obtained by using a skeletal joint detection model; A skeletal joint feature extraction module: the obtained sign language skeletal joint data is regionally divided, and is respectively input into a preset graph convolutional neural network to obtain skeletal joint feature vectors of each region; An RGB image feature extraction module: the sign language RGB image data is input into a ResNet-18 network model based on a multi-dimensional attention module to obtain an RGB image feature vector, and the ResNet-18 model based on the multi-dimensional attention module is constructed by sequentially embedding a spatial attention module and a channel attention module in each residual block in the ResNet-18 network model, so that the network model can pay more attention to the motion information of the hand and arm and remove redundant information; A Bi-LSTM-based time sequence feature extraction module: the feature vectors obtained by the skeletal joint feature extraction module and the RGB image feature extraction module are respectively input into a Bi-LSTM model to obtain sequence features of each mode; A late fusion module: the sequence features of the sign language skeletal joint data and the sequence features of the sign language RGB image data are respectively input into a classification network to obtain sign language prediction results of each mode, and the two prediction results are late fused to obtain a final sign language recognition result.

Images