Method, device and system for sign language recognition based on fusion of visual and skeleton information

Abstract

The application discloses a sign language recognition method, device and system based on fusion of visual and skeleton information, and the sign language recognition method comprises the following steps: inputting obtained sign language video image data and sign language skeleton heat map data into a sampling layer composed of 3D convolution to obtain feature map data; inputting the feature map data into a preset 3D residual convolutional neural network based on multi-dimensional attention enhancement to obtain the spatiotemporal features of the sign language video image and the spatiotemporal features of the sign language skeleton heat map; performing multi-modal sign language action prediction based on the video image features and the skeleton heat map features respectively, and performing late fusion on the prediction outputs of the two modalities to complete the sign language recognition based on fusion of visual and skeleton information. The application can utilize the 3D residual convolutional neural network based on multi-dimensional attention enhancement to represent the spatiotemporal features of the video image and the skeleton heat map respectively, and perform late fusion on the prediction outputs of the two modalities, thereby improving the robustness and accuracy of the sign language recognition result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of sign language recognition, and particularly relates to a sign language recognition method, device and system based on fusion of visual and skeleton information. BACKGROUND

[0002] With the rapid development of video acquisition, human-computer interaction and virtual reality technology, sign language recognition based on video images has attracted more and more attention internationally. Real-time communication of sign language has become an important topic in the field of computer vision and pattern recognition. Since the human skeleton contains rich motion feature information, and the RGB image data contains important human surface texture information, therefore, the relationship between dynamic skeleton modality and motion pattern can be analyzed, or the relationship between dynamic RGB image surface and motion pattern can be analyzed, to realize sign language recognition.

[0003] Although the sign language recognition method based on RGB image and the sign language recognition method based on skeleton have achieved good results in end-to-end sign language recognition, these researches mainly focus on a specific single modality, which often leads to limited generalization ability of the sign language recognition solution. For example: the RGB image is easily affected by light, angle, etc., and the skeleton point data in the skeleton is a sequence of human key points (such as elbow, wrist, knee, etc.), but in the case of difficult access to high-quality key points, the performance of sign language recognition will be greatly affected. Therefore, some important problems are raised: can the information of the two modalities be integrated to take advantage of their complementary advantages in sign language characteristics? And to what extent should different sign language single modality information be processed independently, and what kind of fusion mechanism should be used to obtain the maximum performance gain of the fusion information?

[0004] Effective fusion of multi-modal data helps to make up for the defects of single-modal input, and the robust hand gesture representation can be obtained by utilizing the specificity and complementarity of multiple modalities. Not only the information fusion method of two modalities of skeleton points and RGB images, generally, there are three ways for the fusion of multi-modal information, early fusion, deep fusion and late fusion. Early fusion (Early-Fusion) is to do feature fusion before feature extraction of the original modal data. In theory, this fusion method is probably the best method for multi-modal fusion, because the corresponding features have certain index relationship and less feature abstraction in reality. Deep fusion (Deep Fusion) needs to do certain interaction in the feature layer. For the above two fusion methods, due to the great difference between the data of different modalities, the biggest difference between the skeleton points and the image is the difference in data structure, one is graph data and the other is Euclidean data, and both of them have their own network structure. In the current hand gesture recognition research, the simplest fusion method is not to fuse the features of two modalities in the feature layer or at the beginning, but to use the late fusion strategy. Therefore, we think that the biggest difficulty of information fusion between image and skeleton points is also the problem of data structure difference between pixels and skeleton points. At present, most of the existing deep learning models mainly focus on convolutional neural network (CNN), recurrent neural network (RNN) and graph convolution network (GCN). Among them, CNN and RNN are very suitable for processing Euclidean data such as RGB, depth, optical flow, etc., but they cannot well express the highly nonlinear and complex skeleton point data. GCN is very suitable for processing non-Euclidean skeleton point data, but this method is not friendly to visual image data. Therefore, it is urgent to use a unified deep learning network architecture to integrate the information of two modalities of video image and hand gesture skeleton. SUMMARY

[0005] In order to solve the above problems, the application provides a hand gesture recognition method, device and system based on fusion of visual and skeleton information, which can effectively improve the robustness and recognition accuracy of hand gesture recognition in practical application.

[0006] In order to realize the above technical purpose and achieve the above technical effect, the application realizes the technical scheme as follows:

[0007] In the first aspect, the application provides a hand gesture recognition method based on fusion of visual and skeleton information, comprising:

[0008] The obtained hand gesture video image data and hand gesture skeleton heat map data are respectively input into a first sampling layer and a second sampling layer composed of 3D convolution to obtain first feature map data and second feature map data;

[0009] The first feature map data and the second feature map data are respectively input into a preset 3D residual convolutional neural network based on multi-dimensional attention enhancement to obtain the spatiotemporal features of the sign language video image and the spatiotemporal features of the sign language skeleton heatmap; wherein, the 3D residual convolutional neural network includes two sub-networks with the same structure, each sub-network including a first residual module, a second residual module and a third residual module arranged in sequence, and each residual module is connected to the multi-dimensional attention enhancement module respectively;

[0010] Multimodal sign language movement prediction is performed based on the video image features and skeleton heatmap features, and the prediction outputs of the two modalities are fused late to complete sign language recognition based on the fusion of visual and skeleton information.

[0011] Optionally, the method for acquiring the sign language video image data includes:

[0012] The acquired sign language video is converted into image data to obtain the sign language video image data;

[0013] The method for obtaining the sign language skeleton heatmap data includes:

[0014] The OpenPose library is used to extract sign language-related skeleton information from the sign language video image data, and then it is converted into sign language skeleton heatmap data.

[0015] Optionally, the first residual module consists of L units. The convolution kernel is composed of L, which is a coefficient that controls the number of residual block layers.

[0016] The second residual module consists of L Convolution kernel structure;

[0017] The third residual module consists of L units. Convolutional kernels constitute the structure.

[0018] Optionally, the multi-dimensional attention enhancement module includes a spatial attention module, a temporal attention module, and a channel attention module arranged sequentially;

[0019] The feature map data χ output by the residual module is input into the spatial attention module, which then generates a spatial attention map along the spatial dimension.

[0020] The spatial attention map The feature map data χ is generated by multiplying the residuals into the input feature map data χ. s ;

[0021] The feature map data χ s Perform a residual connection with the feature map data χ to generate feature map data X′. s ;

[0022] The feature map data X' s is input into a time attention module, and a time attention map is generated along a time dimension by the time attention module

[0023] The time attention map is multiplied in a residual manner to the input feature map data X' s , to generate feature map data χ t ;

[0024] The feature map data χ t is connected in a residual manner with the feature map data X' s , to generate feature map data χ' t ;

[0025] The feature map data χ' t is input into a channel attention module, and a channel attention map is generated along a channel dimension by the channel attention module

[0026] The channel attention map is multiplied in a residual manner to the input feature map data χ' t , to generate feature map data χ c ;

[0027] The feature map data χ t is connected in a residual manner with the feature map data χ' t , to generate feature map data χ' c .

[0028] Optionally, the expression of the spatial attention map is:

[0029]

[0030] wherein, represents the feature map data output by the residual module, f is the number of frames of the sign language video image or the sign language skeleton heat map, C is the number of channels of the sign language video image or the sign language skeleton heat map, H and W are the height and width of each frame of the sign language video image or the sign language skeleton heat map, AvgPool t is an average operation on all frames of the sign language video image or the sign language skeleton heat map, that is, represents a 2D convolution along the spatial dimension, the output channel of which is 1, and σ represents an activation function.

[0031] Optionally, the expression of the time attention map is:

[0032]

[0033] wherein, represents the feature map data output by the spatial attention module, T is the frame number of the sign language video image or the sign language skeleton heat map, C is the channel number of the sign language video image or the sign language skeleton heat map, H and W are the height and width of each frame of the sign language video image or the sign language skeleton heat map, AvgPool s represents the average of the feature map data along the spatial dimension, represents the 1D convolution along the time dimension.

[0034] Optionally, the channel attention map The expression is:

[0035]

[0036] wherein, represents the feature map data output by the time module, T is the frame number of the sign language video image or the sign language skeleton heat map, C is the channel number of the sign language video image or the sign language skeleton heat map, H and W are the height and width of each frame of the sign language video image or the sign language skeleton heat map, AvgPool st represents the average of the feature map along the spatial and time dimensions, and are two linear functions running along the channel dimension, and σ is the ReLu activation function.

[0037] Optionally, the multi-modal sign language action prediction is performed based on the video image features and the skeleton heat map features, specifically:

[0038] The video image features and the skeleton heat map features are respectively predicted by using a traditional three-layer fully connected classifier Softmax;

[0039] The fusion formula used for late fusion of the prediction outputs of the two modalities is:

[0040] y 2sPoseRGB = y Pose + α * y RcB

[0041] wherein, y 2sPoseRGB represents the final classification prediction result of the sign language network, y Pose and y RGB respectively represent the prediction results of the skeleton heat map features and the video image features, and α is determined through experimental result feedback and has a value between 0 and 1.

[0042] In a second aspect, the present application provides a sign language recognition device based on fusion of visual and skeleton information, comprising:

[0043] The sign language data acquisition module is configured to input the obtained sign language video image data and sign language skeleton heat map data into a first sampling layer and a second sampling layer respectively formed by 3D convolution, to obtain first feature map data and second feature map data;

[0044] The multi-modal feature extraction module is configured to input the first feature map data and the second feature map data into a preset 3D residual convolutional neural network based on multi-dimensional attention enhancement, to obtain a sign language video image spatio-temporal feature and a sign language skeleton heat map spatio-temporal feature; wherein the 3D residual convolutional neural network comprises two sub-networks with the same structure, each sub-network comprising a first residual module, a second residual module and a third residual module arranged in sequence, and each residual module being connected to a multi-dimensional attention enhancement module.

[0045] The late fusion module is configured to perform multi-modal sign language action prediction based on the video image feature and the skeleton heat map feature respectively, and to perform late fusion on the prediction outputs of the two modalities, to complete sign language recognition based on fusion of visual and skeleton information.

[0046] In a third aspect, the present application provides a sign language recognition system based on fusion of visual and skeleton information, comprising a storage medium and a processor.

[0047] The storage medium is configured to store instructions;

[0048] The processor is configured to operate according to the instructions to perform the method according to any one of the first aspect.

[0049] Compared with the prior art, the present application has the following beneficial effects:

[0050] The present application provides a sign language recognition method, device and system based on fusion of visual and skeleton information, which extracts spatio-temporal features of sign language video image data and sign language skeleton heat map data using a unified neural network architecture, specifically comprising: using a sampling layer formed by 3D convolution to extract first feature map data and second feature map data from the sign language video image data and sign language skeleton heat map data respectively, using a 3D residual convolutional neural network based on multi-dimensional attention enhancement to extract spatio-temporal features of the first feature map data and the second feature map data, and a multi-dimensional attention enhancement mechanism can focus on key feature information of sign language video images and sign language skeleton heat maps in space-time-channel from different dimensions. Then, multi-modal sign language action prediction is performed based on the video image feature and the skeleton heat map feature respectively, and late fusion is performed on the prediction outputs of the two modalities, to complete fusion of the sign language video information and the skeleton information, effectively improving the robustness and recognition accuracy of sign language recognition in practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor, wherein:

[0052] Figure 1 is a model structure diagram of a sign language recognition method based on fusion of visual and skeleton information in an embodiment of the present application;

[0053] Figure 2 is a schematic diagram of a multi-dimensional attention enhancement module used in a sign language recognition method based on fusion of visual and skeleton information in an embodiment of the present application;

[0054] Figure 3 is a structural block diagram of a sign language recognition device embodiment based on fusion of visual and skeleton information in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0056] In addition, if the present application embodiments involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0057] Embodiment 1

[0058] A sign language recognition method based on fusion of visual and skeleton information is provided in the embodiments of the present application, which includes the following steps:

[0059] (1) input the obtained sign language video image data and sign language skeleton heat map data into a first sampling layer and a second sampling layer composed of 3D convolution respectively, to obtain first feature map data and second feature map data;

[0060] (2) input the first feature map data and the second feature map data into a preset 3D residual convolutional neural network based on multi-dimensional attention enhancement respectively, to obtain spatiotemporal features of sign language video image spatiotemporal features and sign language skeleton heat map; wherein the 3D residual convolutional neural network comprises two subnetworks with the same structure, each subnetwork comprises a first residual module, a second residual module and a third residual module arranged in sequence, and each residual module is connected with a multi-dimensional attention enhancement module;

[0061] (3) based on the video image features and the skeleton heat map features, multi-modal sign language action prediction is performed respectively, and late fusion is performed on the prediction outputs of the two modalities, to complete sign language recognition based on fusion of visual and skeleton information.

[0062] In the embodiment of the application, the unified neural network architecture is used to extract the spatiotemporal features of the sign language video image data and the sign language skeleton heat map data, specifically, the first feature map data and the second feature map data are extracted from the sign language video image data and the sign language skeleton heat map data by using a sampling layer composed of 3D convolution, and the spatiotemporal features of the first feature map data and the second feature map data are extracted by using the 3D residual convolutional neural network based on multi-dimensional attention enhancement. The multi-dimensional attention enhancement mechanism can focus on the key feature information of the sign language video image and the sign language skeleton heat map in the space-time-channel from different dimensions. Then, based on the video image features and the skeleton heat map features, multi-modal sign language action prediction is performed respectively, and late fusion is performed on the prediction outputs of the two modalities, to complete the fusion of the sign language video information and the skeleton information, effectively improving the robustness and recognition accuracy of sign language recognition in practical application.

[0063] In one specific embodiment of the embodiment of the application, the method for obtaining the sign language video image data comprises:

[0064] The obtained sign language video is converted into image data to obtain the sign language video image data;

[0065] The method for obtaining the sign language skeleton heat map data comprises:

[0066] The sign language related skeleton information is obtained from the sign language video image data by using the OpenPose library, and then converted into the sign language skeleton heat map data.

[0067] In one specific embodiment of the embodiment of the application, the first residual module is composed of L convolutional kernels, which are used for spatiotemporal convolution operation, and L is a coefficient for controlling the number of residual block layers; The second residual module is composed of L convolutional kernels;

[0068]

[0069] ​​The third residual module consists of L units. Convolutional kernels constitute the structure.

[0070] In one specific embodiment of the present invention, the multi-dimensional attention enhancement module includes a spatial attention module, a temporal attention module, and a channel attention module arranged sequentially;

[0071] The feature map data χ output by the residual module is input into the spatial attention module, which then generates a spatial attention map along the spatial dimension.

[0072] The spatial attention map The feature map data χ is generated by multiplying the residuals into the input feature map data χ. s ;

[0073] The feature map data χ s Perform a residual join with the feature map data χ to generate feature map data χ. s ′;

[0074] feature map data χ s The input is fed into the time attention module, which then generates a time attention map along the time dimension.

[0075] The time attention map Multiply the residual by the input feature map data X′ s Generate feature map data χ t ;

[0076] The feature map data χ t With feature map data X′ s Perform residual connections to generate feature map data χ′ t ;

[0077] The feature map data χ′ t The input is fed into the channel attention module, which then generates a channel attention map along the channel dimension.

[0078] The channel attention map Multiply the residual by the input feature map data χ′ t Generate feature map data χ c ;

[0079] The feature map data χ t With feature map data χ′ t Perform residual connections to generate feature map data χ′ c .

[0080] Among them, the spatial attention map The expression is:

[0081]

[0082] in, This represents the feature map data output by the residual module. T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool t Averaging is performed on all frames, i.e. σ represents a 2D convolution along the spatial dimension, with 1 output channel, and σ represents an activation function.

[0083] The time attention map The expression is:

[0084]

[0085] in, The representation is the feature map data output by the spatial attention module, where T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool s This indicates averaging the feature map data along the spatial dimension. This represents a 1D convolution along the time dimension.

[0086] The channel attention map The expression is:

[0087]

[0088] in, This represents the feature map data output by the time attention module. T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool st This indicates averaging the feature maps along the spatial and temporal dimensions. and These are two linear functions that run along the channel dimension, where σ is the ReLU activation function.

[0089] In one specific embodiment of the present invention, the step of performing multimodal sign language movement prediction based on the video image features and the skeleton heatmap features respectively is as follows:

[0090] The video image features and the skeleton heat map features are respectively predicted by using a traditional three-layer full connection classifier Softmax;

[0091] The fusion formula used for late fusion of the prediction outputs of the two modalities is:

[0092] y 2sPoseRGB = y Pose + a * y RGB

[0093] wherein y 2sPoseRGB represents the final classification prediction result of the sign language network, y Pose and y RGB respectively represent the prediction results of the skeleton heat map features and the video image features, and a is an experimental result feedback value between 0 and 1.

[0094] The method in the embodiment of the present application will be described in detail below in combination with a specific implementation.

[0095] As shown in Figure 1 , the sign language recognition method based on fusion of visual and skeleton information proposed in the embodiment of the present application is implemented according to the following steps:

[0096] Step 1, obtain sign language video image data and sign language skeleton heat map data, and input the obtained sign language video image data and sign language skeleton heat map data into a first sampling layer and a second sampling layer respectively composed of 3D convolution, to obtain first feature map data and second feature map data;

[0097] Step 2, input the first feature map data and the second feature map data into a preset 3D residual convolutional neural network based on multi-dimensional attention enhancement, to obtain sign language video image spatio-temporal features and sign language skeleton heat map spatio-temporal features; wherein the 3D residual convolutional neural network includes two subnetworks with the same structure, each subnetwork includes a first residual module, a second residual module and a third residual module arranged in sequence, and each residual module is connected with a multi-dimensional attention enhancement module, the multi-dimensional attention enhancement module has an attention enhancement mechanism and can focus on key feature information of the sign language video image and the sign language skeleton heat map in space-time-channel from different dimensions;

[0098] Step 3, perform multi-modal sign language action prediction based on the video image features and the skeleton heat map features respectively, and perform late fusion of the prediction outputs of the two modalities, to realize fusion of the sign language video information and the skeleton information, and complete sign language recognition based on fusion of visual and skeleton information.

[0099] Referring to step 1 above, obtain the sign language video and convert it into image data. Each image is 56×56 in size. Therefore, the dimension of the obtained sign language video image T-frame data is 3×T×56×56.

[0100] The sign language-related skeleton information is obtained from the sign language video images using the OpenPose library and then converted into sign language skeleton heatmap data. Specifically, the resolution of all sign language videos needs to be adjusted to a smaller value (340x256), and the conversion frame rate is 30FPS. This aims to improve the accuracy of pose evaluation and reduce the model's inference time. Next, the OpenPose tool is used to estimate the positions of 28 joints directly related to the sign language in each frame. This toolbox provides 2D coordinates (x, y) in a pixel coordinate system and gives the confidence scores c for 18 human joints. After extracting the 2D skeleton data from the sign language video images, it is reformulated as a 3D heatmap stack. Formally, a 2D pose is represented as a heatmap of size K×H×W, where K is the number of joints, and H and W are the height and width of each frame. The triplet coordinates of the skeleton joints are (x, y)... k ,y k ,c k We can obtain the skeleton heatmap G by synthesizing K Gaussian maps at each joint:

[0101]

[0102] σ controls the variance of the Gaussian mapping, (x k ,y k ) and c k These represent the position and confidence score of the k-th node, respectively. Finally, a 3D heatmap stack is obtained by stacking all heatmaps G along the time dimension, so its size is K×T×H×W, as shown below. Figure 1 The skeleton heatmap data shown is as follows. Therefore, the dimensions of the obtained sign language skeleton heatmap data are 28×T×56×56.

[0103] The sampling layer employs a 3D convolutional layer to sample data from different modalities. For a video image of T frames, with a size dimension of 3×T×56×56, the sampling layer yields input data of 32×T×56×56 for the 3D residual convolutional neural network based on multi-dimensional attention enhancement. For a T-frame skeleton heatmap sequence of 28×T×56×56, the sampling layer provides input data of 32×T×56×56 for the model.

[0104] Referring to the above step 2, the first feature map data and the second feature map data are respectively input into a preset 3D residual convolutional neural network based on multi-dimensional attention enhancement to obtain spatiotemporal features of the sign language video image and spatiotemporal features of the sign language skeleton heat map; wherein the 3D residual convolutional neural network comprises two subnetworks of the same structure, each subnetwork comprises a first residual module, a second residual module and a third residual module arranged in sequence, and each residual module is connected with a multi-dimensional attention enhancement module, the multi-dimensional attention enhancement module has an attention enhancement mechanism and can focus on key feature information of the sign language video image and the sign language skeleton heat map in the space-time-channel from different dimensions, and the specific steps are as follows:

[0105] (1) The residual module Res3D_1 composed of L convolutional kernels is used to perform spatiotemporal convolution operation. Here L represents a coefficient for controlling the number of residual block layers, and the size is 4. After this processing, the size of the feature map is 128xTx28x28.

[0106] (2) The multi-dimensional attention enhancement module follows the residual module Res3D_1, as shown in Figure 2 . The attention mechanism has many forms, including spatial attention, temporal attention and channel attention, etc., which effectively capture the dynamic spatiotemporal correlation in the video image data from different aspects. Here, we use the multi-dimensional attention enhancement module as shown in Figure 2 . It includes three sub-modules: spatial attention module, temporal attention module and channel attention module. The input feature map is input into the three modules in order, and the three modules generate attention maps along different dimensions. Then these attention maps are multiplied with the input feature map to enhance the corresponding features. Residual connections are added to all attention modules to stabilize the training. Here, spatial attention helps to model and strengthen the features of image texture, which are important for describing certain hand gesture actions. The calculation is as follows:

[0107]

[0108] Here, represents the feature map output by the residual module, T is the number of frames of the sign language video image or the sign language skeleton heat map, C is the number of channels of the sign language video image or the sign language skeleton heat map, H and W are the height and width of each frame of the sign language video image or the sign language skeleton heat map. AvgPool t The average operation is performed on all frames, i.e. represents a 2D convolution along the spatial dimension, and the output channel is 1. σ represents an activation function. Then, the spatial attention map is multiplied with in a residual manner. ​to the input feature map χ for adaptive feature refinement and adding residual connection for spatial attention for stabilizing the training of its network. It is computed as follows:

[0109]

[0110]

[0111] Temporal attention is similar to spatial attention. It helps the model to pay more attention to the frames that are in more important stages of sign language recognition. It is computed as follows:

[0112]

[0113] where denotes the feature map data output by the spatial attention module, T is the number of frames of the sign language video image or the sign language skeleton heat map, C is the number of channels of the sign language video image or the sign language skeleton heat map, H and W are the height and width of each frame of the sign language video image or the sign language skeleton heat map, AvgPool s denotes the average of the feature map along the spatial dimension. denotes the 1D convolution along the temporal dimension. Then, the temporal attention map is point multiplied in a residual manner to the input feature map χ s for adaptive feature refinement and adding residual connection for temporal attention for stabilizing the training of its network. It is computed as follows:

[0114]

[0115]

[0116] Channel attention helps to strengthen the descriptive features (channels) according to the input samples. The attention map it generates is as follows:

[0117]

[0118] where, denotes the feature map data output by the temporal attention module, T is the number of frames of the sign language video image or the sign language skeleton heat map, C is the number of channels of the sign language video image or the sign language skeleton heat map, H and W are the height and width of each frame of the sign language video image or the sign language skeleton heat map, AvgPool st denotes the average of the feature map along the spatial and temporal dimensions. and are two linear functions running along the channel dimension. σ is the ReLu activation function. Then, the channel attention map Residual way dot product to input feature map χ t ′ for adaptive feature refinement, and add a residual connection for channel attention for stabilizing the training of its network. The calculation is as follows:

[0119]

[0120]

[0121] Therefore, after this multi-dimensional attention enhancement operation, the obtained attention feature map χ′ c is input as the third layer of the model.

[0122] (3) For the feature map of the third layer, a residual module Res3D_2 with L = 6 is used, Res3D_2 is mainly composed of convolution kernels, and the output feature map obtained is 256 × T × 14 × 14.

[0123] (4) After the 3D convolution operation, the multi-dimensional attention enhancement operation described above is performed on each feature map, and the obtained attention feature map is input as the fourth layer of the model.

[0124] (5) For the feature map of the third layer, a residual module Res3D_3 with L = 3 is used, Res3D_3 is mainly composed of convolution kernels, and the output feature map obtained is 256 × T × 7 × 7.

[0125] (6) After the third 3D residual convolution operation, the multi-dimensional attention enhancement operation described above is performed on each feature map, and the obtained attention feature map is input as the video image spatiotemporal feature or skeleton heat map spatiotemporal feature of the model.

[0126] Referring to the above step 3, the video image feature and the skeleton heat map feature are respectively subjected to multi-modal sign language action prediction, and then the prediction outputs of the two modalities are fused to realize the fusion of the sign language video information and the skeleton information. Specifically, a traditional three-layer fully connected classifier Softmax is used to predict the sign language action of the video image feature and the skeleton heat map feature. Then, a late fusion strategy is used to fuse the skeleton information and the image information, and the specific implementation is as follows:

[0127] y 2sPoseRGB = y Pose + α * y RGB

[0128] wherein y 2sPoseRGB represents the final classification prediction result of the sign language network, y Pose and yRGB These represent the prediction results for skeleton heatmap features and video image features, respectively. α is an adjustable hyperparameter based on experimental results, with a value between 0 and 1.

[0129] In this specific embodiment, the feasibility of the present invention is verified by referring to specific examples:

[0130] Experiments were conducted using the Chinese DEVISIGN-D dataset and the American ASLLVD dataset for validation. The DEVISIGN-D dataset is a multimodal sign language dataset containing 500 commonly used words. This dataset includes three modalities: RGB video, 3D skeleton, and depth sequence. RGB video data acquired using a Kinect V2 sensor was used for sign language skeleton extraction, forming a stacked skeleton heatmap suitable for sign language recognition for the experiments. The ASLLVD dataset is a language RGB video dataset from the American Sign Language Dictionary, containing a number and type of symbols similar in size and scope to the vocabulary entries in existing English-to-ASL dictionaries. Twenty dynamic gestures from RGB video were randomly selected from the ASLLVD dataset as the experimental dataset. Due to the small sample size (1080 samples in total) generated in this process, a new dataset splitting strategy was necessary. Using 3-fold cross-validation, the training set of ASLLVD was set to 720 samples, and the test set to 360 samples. This strategy improved the balance of sample size to adequately evaluate the model. The test set is used during the testing phase after the entire training task is completed, and its output represents the final recognition accuracy.

[0131] Based on the comparative analysis of different strategies employed by our proposed method on the DEVISIGN-D and ASLLVD datasets shown in Tables 1 and 2, it is evident that: for validation across different modalities, the multimodal fusion method outperforms the single-modal method. For single-modal validation, skeleton data typically outperforms video image data in terms of recognition accuracy. One possible explanation is that the information in the skeleton data is more robust, as it benefits from training with massive and highly variable data. On the other hand, since video image data relies on the original data and learns only from the training set, some overfitting may occur. Furthermore, by combining skeleton data and video image data, it yields the expected significant improvement.

[0132] Table 1. Experimental results of various models and fusion frameworks on DEVISIGN-D.

[0133]

[0134] Table 2. Experimental results of various models and fusion frameworks on ASLLVD.

[0135]

[0136] According to the comparison of the results of the method model and the existing methods on the dataset ASLLVD shown in Table 3, it can be observed that the performance of the deep learning method (such as ST-GCN, AGCN and Aegles) is better than that of the artificial feature method (such as MEI, MHI and PCA), but not better than that of the multi-modal fusion method, i.e., the model provided in the present application. Specifically, compared with the baselines of BHOF and Aegles, the recognition rate of the method model in the present application embodiment is increased by 8.79% and 5.91% respectively, which indicates that the method proposed in the present application embodiment achieves a higher accuracy, and further illustrates the feasibility, advancement and superiority of the method in the present application embodiment.

[0137] Table 3 Recognition results of the method of the present application embodiment and other disclosed methods on ASLLVD

[0138]

[0139] In the present application embodiment, a sign language recognition method based on fusion of visual information and skeleton information is provided. Unlike the fusion of traditional skeleton data and video image data, the method in the present application uses human skeleton point coordinates to generate 3D heat map stacks to form skeleton heat map data, extracts skeleton heat map spatio-temporal features and video image spatio-temporal features using a unified 3D residual convolutional network model based on multi-dimensional attention enhancement, and dynamically selects important information in space, time and channel by introducing different dimensional attention mechanisms to focus on video sign language skeleton information and image information in different dimensions. Finally, the sign language prediction outputs of the video image spatio-temporal features and the skeleton heat map spatio-temporal features are fused to obtain the final result of sign language recognition. Experimental results show that the recognition rate of the method of the present application on the DEVISIGN-D and ASLLVD datasets can reach 85.40% and 93.79% respectively. This result verifies the effectiveness of the method of the present application for fusion based on video image information and skeleton information, and improves the problem of insufficient generalization ability of single modal. In summary, in the task of sign language recognition based on fusion of video image information and skeleton information, the method improves the robustness and accuracy of sign language recognition results, and is particularly suitable for complex and variable sign language recognition.

[0140] Embodiment 2

[0141] Based on the same inventive concept as in Embodiment 1, as shown in Figure 3 In the present application embodiment, a sign language recognition device based on fusion of visual and skeleton information is provided, which comprises.

[0142] The sign language data acquisition module is configured to input the obtained sign language video image data and sign language skeleton heat map data into a first sampling layer and a second sampling layer respectively formed by 3D convolution, to obtain first feature map data and second feature map data;

[0143] The multi-modal feature extraction module is configured to input the first feature map data and the second feature map data into a preset 3D residual convolutional neural network based on multi-dimensional attention enhancement, to obtain a sign language video image spatiotemporal feature and a sign language skeleton heat map spatiotemporal feature; wherein the 3D residual convolutional neural network comprises two sub-networks of the same structure, each of the sub-networks comprises a first residual module, a second residual module and a third residual module arranged in sequence, and each of the residual modules is connected with a multi-dimensional attention enhancement module;

[0144] The late fusion module is configured to perform multi-modal sign language action prediction based on the video image feature and the skeleton heat map feature respectively, and perform late fusion on the prediction outputs of the two modalities, to complete sign language recognition based on fusion of visual and skeleton information.

[0145] Embodiment 3

[0146] Based on the same inventive concept as in Embodiment 1, a sign language recognition system based on fusion of visual and skeleton information is provided in the embodiments of the present application, comprising a storage medium and a processor.

[0147] The storage medium is configured to store instructions.

[0148] The processor is configured to operate according to the instructions to perform the method according to any one of Embodiments 1.

[0149] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0150] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions described in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer-usable storage medium that can guide the computer-usable storage medium to operate in such a way that the computer program instructions stored in the computer-usable storage medium produce a device that implements the functions described in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart

[0151] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart

[0153] The above embodiments of the present application have been described with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and the above-described embodiments are merely illustrative, not restrictive, and those skilled in the art can make many modifications without departing from the spirit and scope of the present application, and these modifications are also included in the scope of the present application.

[0154] The basic principles and main features of the present application and the advantages of the present application have been shown and described above. It should be understood by those skilled in the art that the present application is not limited to the above-described embodiments, and the above-described embodiments and descriptions in the specification are merely illustrative of the principles of the present application, and various changes and modifications can be made without departing from the spirit and scope of the present application, and these changes and modifications are also included in the scope of the present application. The scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A sign language recognition method based on the fusion of visual and skeletal information, characterized in that, include: The acquired sign language video image data and sign language skeleton heatmap data are respectively input into the first sampling layer and the second sampling layer composed of 3D convolution to obtain the first feature map data and the second feature map data; The first feature map data and the second feature map data are respectively input into a preset 3D residual convolutional neural network based on multi-dimensional attention enhancement to obtain the spatiotemporal features of the sign language video image and the spatiotemporal features of the sign language skeleton heatmap; wherein, the 3D residual convolutional neural network includes two sub-networks with the same structure, each sub-network including a first residual module, a second residual module and a third residual module arranged in sequence, and each residual module is connected to the multi-dimensional attention enhancement module respectively; Multimodal sign language action prediction is performed based on the video image features and skeleton heatmap features, and the prediction outputs of the two modalities are fused late to complete sign language recognition based on the fusion of visual and skeleton information. The multi-dimensional attention enhancement module includes a spatial attention module, a temporal attention module, and a channel attention module arranged in sequence. The feature map data χ output by the residual module is input into the spatial attention module, which then generates a spatial attention map along the spatial dimension. The spatial attention map The feature map data χ is generated by multiplying the residuals into the input feature map data χ. s ; The feature map data χ s Perform a residual join with the feature map data χ to generate feature map data χ′. s ; The feature map data χ′ s The input is fed into the time attention module, which then generates a time attention map along the time dimension. The time attention map Multiply the residual by the input feature map data χ′ s Generate feature map data χ t ; The feature map data χ t With feature map data χ′ s Perform residual connections to generate feature map data χ′ t ; The feature map data χ′ t The input is fed into the channel attention module, which then generates a channel attention map along the channel dimension. The channel attention map Multiply the residual by the input feature map data χ′ t Generate feature map data χ c ; The feature map data χ t With feature map data χ′ t Perform residual connections to generate feature map data χ′ c ; The spatial attention map The expression is: in, This represents the feature map data output by the residual module. T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool t Averaging is performed on all frames of sign language video images or sign language skeleton heatmaps, i.e. This represents a 2D convolution along the spatial dimension, with one output channel, and σ represents an activation function. The time attention map The expression is: in, The representation is the feature map data output by the spatial attention module, where T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool s This indicates averaging the feature map data along the spatial dimension. This represents a 1D convolution along the time dimension; The channel attention map The expression is: in, This represents the feature map data output by the temporal module. T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool st This indicates averaging the feature maps along the spatial and temporal dimensions. and These are two linear functions that run along the channel dimension, where σ is the ReLU activation function.

2. The sign language recognition method based on visual and skeleton information fusion according to claim 1, characterized in that: The method for acquiring the sign language video image data includes: The acquired sign language video is converted into image data to obtain the sign language video image data; The method for obtaining the sign language skeleton heatmap data includes: The OpenPose library is used to extract sign language-related skeleton information from the sign language video image data, and then it is converted into sign language skeleton heatmap data.

3. The sign language recognition method based on visual and skeleton information fusion according to claim 1, characterized in that: The first residual module consists of L units The convolution kernel is composed of L, which is a coefficient that controls the number of residual block layers. The second residual module consists of L Convolution kernel structure; The third residual module consists of L units. Convolutional kernels constitute the structure.

4. The sign language recognition method based on visual and skeleton information fusion according to claim 1, characterized in that: The step of predicting multimodal sign language actions based on the video image features and skeleton heatmap features is specifically: using a traditional three-layer fully connected classifier Softmax to predict sign language actions based on the video image features and skeleton heatmap features respectively; The fusion formula used for late fusion of the prediction outputs of these two modes is as follows: and 2sPoseRGB / and Pose +α*y RGB Among them, y 2sPoseRGB y represents the final classification prediction result of the sign language network. Pose and y RGB These represent the prediction results of skeleton heatmap features and video image features, respectively. α is the feedback from the experimental results, and its value is between 0 and 1.

5. A sign language recognition device based on the fusion of visual and skeletal information, characterized in that, include: The sign language data acquisition module is used to input the acquired sign language video image data and sign language skeleton heat map data into the first sampling layer and the second sampling layer composed of 3D convolution, respectively, to obtain the first feature map data and the second feature map data. A multimodal feature extraction module is used to input the first feature map data and the second feature map data into a preset 3D residual convolutional neural network based on multidimensional attention enhancement, respectively, to obtain the spatiotemporal features of the sign language video image and the spatiotemporal features of the sign language skeleton heatmap; wherein, the 3D residual convolutional neural network includes two sub-networks with the same structure, each sub-network including a first residual module, a second residual module and a third residual module arranged in sequence, and each residual module is connected to the multidimensional attention enhancement module; The late fusion module is used to predict multimodal sign language actions based on the video image features and skeleton heatmap features respectively, and to perform late fusion on the prediction outputs of the two modalities to complete sign language recognition based on the fusion of visual and skeleton information. The multi-dimensional attention enhancement module includes a spatial attention module, a temporal attention module, and a channel attention module arranged in sequence. The feature map data χ output by the residual module is input into the spatial attention module, which then generates a spatial attention map along the spatial dimension. The spatial attention map The feature map data χ is generated by multiplying the residuals into the input feature map data χ. s ; The feature map data χ s Perform a residual join with the feature map data χ to generate feature map data χ′. s ; The feature map data χ′ s The input is fed into the time attention module, which then generates a time attention map along the time dimension. The time attention map Multiply the residual by the input feature map data χ′ s Generate feature map data χ t ; The feature map data χ t With feature map data χ′ s Perform residual connections to generate feature map data χ′ t ; The feature map data χ′ t The input is fed into the channel attention module, which then generates a channel attention map along the channel dimension. The channel attention map Multiply the residual by the input feature map data χ′ t Generate feature map data χ c ; The feature map data χ t With feature map data χ′ t Perform residual connections to generate feature map data χ′ c ; The spatial attention map The expression is: in, This represents the feature map data output by the residual module. T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool t Averaging is performed on all frames of sign language video images or sign language skeleton heatmaps, i.e. This represents a 2D convolution along the spatial dimension, with one output channel, and σ represents an activation function. The time attention map The expression is: in, The representation is the feature map data output by the spatial attention module, where T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool s This indicates averaging the feature map data along the spatial dimension. This represents a 1D convolution along the time dimension; The channel attention map The expression is: in, This represents the feature map data output by the temporal module. T is the number of frames in the sign language video image or sign language skeleton heatmap, C is the number of channels in the sign language video image or sign language skeleton heatmap, and H and W are the height and width of each frame of the sign language video image or sign language skeleton heatmap. AvgPool st This indicates averaging the feature maps along the spatial and temporal dimensions. and These are two linear functions that run along the channel dimension, where σ is the ReLU activation function.

6. A sign language recognition system based on the fusion of visual and skeletal information, characterized in that, Including storage media and processor; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the method according to any one of claims 1-4.

Images