Sign language recognition method and device based on spatiotemporal cross attention network based on full body features

Abstract

The present invention discloses a method and device for sign language recognition based on a spatiotemporal cross-attention network of full-body features, the method comprising: inputting an RGB video into a full-body posture estimator to obtain the target position coordinates of key points of the human body; generating an initial heat map of the skeleton according to the target position coordinates; generating a first original feature and a first interactive feature according to the RGB video; generating a second original feature and a second interactive feature according to the initial heat map; inputting the first original feature, the first interactive feature, the second original feature and the second interactive feature into a spatiotemporal cross-attention module for optimization processing to obtain a first optimized feature and a second optimized feature; performing a splicing operation on the first optimized feature and the second optimized feature to obtain a fused feature; inputting the fused feature into a classifier to obtain a sign language recognition result. The present invention can improve the utilization rate of key points of the human body and the accuracy of sign language recognition, and can be widely used in the technical field of sign language recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of sign language recognition, and in particular to a sign language recognition method and device based on spatiotemporal cross-attention network with full-body features. Background Art

[0002] Among the existing sign language recognition methods, some methods only consider hand movement features, and some methods only extract facial features and hand movement features. There are also a few studies that consider limb features, facial features, and hand features, but the popular graph convolutional network sign language recognition method will bring high model complexity and unexpected noise due to too many key points and edges. In recent years, some studies have also proposed methods to reduce the whole body skeleton graph to retain a small number of key points, but this is not enough to recognize gestures with complex gestures and movements. Features involving non-hand parts, such as facial micro-expressions and lip movements, will be misrecognized due to the small number of nodes, reducing the accuracy of sign language recognition. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a spatiotemporal cross-attention network sign language recognition method and device based on full-body features to improve the accuracy of sign language recognition.

[0004] To achieve the above object, an embodiment of the present invention provides a method for sign language recognition based on spatiotemporal cross attention network based on full-body features, the method comprising:

[0005] Input the RGB video into the full body posture estimator to obtain the target position coordinates of the key points of the human body;

[0006] Generating an initial heat map of the skeleton according to the target position coordinates;

[0007] Generate a first original feature and a first interactive feature according to the RGB video;

[0008] Generate a second original feature and a second interactive feature according to the initial heat map;

[0009] Inputting the first original feature, the first interactive feature, the second original feature, and the second interactive feature into a spatiotemporal cross attention module for optimization processing to obtain a first optimized feature and a second optimized feature;

[0010] Performing a splicing operation on the first optimized feature and the second optimized feature to obtain a fusion feature;

[0011] The fused features are input into a classifier to obtain a sign language recognition result.

[0012] In some embodiments, the step of inputting the RGB video into a full-body posture estimator to obtain target position coordinates of key points of the human body includes:

[0013] Extracting the key points of the upper body skeleton in the RGB video and the key points of the lower body skeleton in the RGB video by the whole body posture estimator;

[0014] Obtaining first position coordinates of key points of the upper body skeleton;

[0015] The key point of the human body of the lower body skeleton is set as the origin to obtain the second position coordinate.

[0016] In some embodiments, before generating the second original feature and the second interactive feature, the method further includes:

[0017] The initial heat map is modeled with the target position coordinates of each of the key points of the human body as the center to obtain a Gaussian heat map.

[0018] In some embodiments, the inputting the first original feature, the first interactive feature, the second original feature, and the second interactive feature into the spatiotemporal cross attention module for optimization processing to obtain the first optimized feature and the second optimized feature includes:

[0019] generating a first query vector, a first key vector, and a first value vector according to the first interaction feature;

[0020] Obtaining a global space template graph according to the first key vector and the first value vector;

[0021] generating a second query vector, a second key vector, and a second value vector according to the second interaction feature;

[0022] Obtaining a global context template graph according to the second key vector and the second value vector;

[0023] Obtaining a first feature map according to the global space template map and the second query vector;

[0024] Obtaining a second feature graph according to the global context template graph and the first query vector;

[0025] Aggregating the first feature map with the first original feature to obtain the first optimized feature;

[0026] The second feature map is aggregated with the second original feature to obtain the second optimized feature.

[0027] In some embodiments, obtaining a first feature map according to the global space template map and the second query vector includes:

[0028] Preset zoom factor;

[0029] Normalizing the global space template map according to the scaling factor and the softmax function;

[0030] The normalized global space template map is mapped to the second query vector to obtain the first feature map.

[0031] In some embodiments, obtaining a second feature map according to the global context template map and the first query vector includes:

[0032] Preset zoom factor;

[0033] Normalizing the global context template graph according to the scaling factor and the softmax function;

[0034] The normalized global context template map is mapped to the first query vector to obtain the second feature map.

[0035] To achieve the above purpose, another aspect of an embodiment of the present invention provides a spatiotemporal cross-attention network sign language recognition device based on full-body features, the device comprising:

[0036] The first module is used to input the RGB video into the whole body posture estimator to obtain the target position coordinates of the key points of the human body;

[0037] The second module is used to generate an initial heat map of the skeleton according to the target position coordinates;

[0038] A third module is used to generate a first original feature and a first interactive feature according to the RGB video;

[0039] A fourth module is used to generate a second original feature and a second interactive feature according to the initial heat map;

[0040] A fifth module, configured to input the first original feature, the first interactive feature, the second original feature, and the second interactive feature into a spatiotemporal cross attention module for optimization processing to obtain a first optimized feature and a second optimized feature;

[0041] A sixth module is used to perform a splicing operation on the first optimization feature and the second optimization feature to obtain a fusion feature;

[0042] The seventh module is used to input the fusion features into a classifier to obtain a sign language recognition result.

[0043] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned spatiotemporal cross-attention network sign language recognition method based on full-body features.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned spatiotemporal cross-attention network sign language recognition method based on full-body features.

[0045] To achieve the above-mentioned purpose, another aspect of the embodiment of the present invention provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the aforementioned spatiotemporal cross-attention network sign language recognition method based on full-body features.

[0046] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a method and device for sign language recognition based on full-body features of a spatiotemporal cross-attention network. The scheme inputs an RGB video into a full-body posture estimator to obtain the target position coordinates of key points of the human body; generates an initial heat map of the skeleton according to the target position coordinates; generates a first original feature and a first interactive feature according to the RGB video; generates a second original feature and a second interactive feature according to the initial heat map; inputs the first original feature, the first interactive feature, the second original feature and the second interactive feature into a spatiotemporal cross-attention module for optimization processing to obtain a first optimized feature and a second optimized feature; performs a splicing operation on the first optimized feature and the second optimized feature to obtain a fused feature; inputs the fused feature into a classifier to obtain a sign language recognition result. The present invention represents the skeleton of the whole body in the form of a heat map, improves the utilization rate of key points of the human body, and reduces the complexity of the model. At the same time, the present invention only uses RGB video and a heat map of the skeleton, which reduces the complexity of the model and improves the accuracy of sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 is a flow chart of a method for sign language recognition based on spatiotemporal cross-attention network of full-body features provided in an embodiment of the present application;

[0049] Figure 2 It is a schematic diagram of the overall architecture of the model provided in the embodiment of the present application;

[0050] Figure 3 is a schematic diagram showing key points of the human body before correction provided in an embodiment of the present application;

[0051] Figure 4 is a schematic diagram of a modified display of key points of a human body provided in an embodiment of the present application;

[0052] Figure 5 It is a schematic diagram of the process of generating a heat map provided in an embodiment of the present application;

[0053] Figure 6 It is a schematic diagram of the structure of the spatiotemporal cross attention module provided in an embodiment of the present application;

[0054] Figure 7 It is a structural schematic diagram of a spatiotemporal cross-attention network sign language recognition device based on full-body features provided in an embodiment of the present application;

[0055] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0057] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0058] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] Among the existing sign language recognition methods, the popular graph convolutional network sign language recognition method will bring high model complexity and unexpected noise due to too many key points and edges. Some studies have also proposed methods to reduce the whole body skeleton graph to retain a small number of key points, but this is not enough to recognize gestures with complex movements. Features involving non-hand parts, such as facial micro-expressions and lip movements, will be misidentified due to the small number of nodes, reducing the accuracy of sign language recognition. In order to improve the accuracy of sign language recognition, some studies have explored multimodal sign language recognition, that is, obtaining multimodal data from different sources, including skeleton, RGB, optical flow and depth data. Different modalities can provide complementary information, and more powerful reasoning results can be obtained by integrating multiple modalities. Most scholars use late fusion to weightedly integrate the classification outputs obtained from each modality to improve the overall performance. However, these methods still have some limitations. First, the various modalities are still independent, lacking further exploration of their complementarity, and no information interaction between each other in the feature extraction stage, resulting in insufficient utilization of human body features. Second, different features and different representations between modalities may lead to suboptimal interaction effects and recognition performance. Third, due to the complexity of acquiring multimodal data and the limited availability of relevant datasets, methods that include multiple modalities have not yet been applied to real-time sign language recognition systems.

[0061] In view of this, if Figure 1 As shown, the embodiment of the present invention provides a time-space cross-attention network sign language recognition method based on full-body features. Figure 1 The method may include but is not limited to steps S100 to S700.

[0062] Step S100, inputting the RGB video into a whole body posture estimator to obtain the target position coordinates of key points of the human body;

[0063] Step S200, generating an initial heat map of the skeleton according to the target position coordinates;

[0064] Step S300, generating a first original feature and a first interactive feature according to the RGB video;

[0065] Step S400, generating a second original feature and a second interactive feature according to the initial heat map;

[0066] Step S500, inputting the first original feature, the first interactive feature, the second original feature, and the second interactive feature into a spatiotemporal cross attention module for optimization processing to obtain a first optimized feature and a second optimized feature;

[0067] Step S600, performing a splicing operation on the first optimization feature and the second optimization feature to obtain a fusion feature;

[0068] Step S700: input the fusion feature into a classifier to obtain a sign language recognition result.

[0069] Steps S100 to S700 shown in the embodiment of the present invention construct an improved spatiotemporal cross-attention module, which takes RGB video and skeleton heat map as input, wherein the RGB video contains rich visual information, and the skeleton captures detailed information of human joints, thereby realizing feature-level interaction between RGB video modal features and skeleton modal features, improving the utilization rate of each modal feature, and also improving recognition efficiency and recognition accuracy.

[0070] In some embodiments, step S100 may include but is not limited to steps S110 to S130:

[0071] Step S110, extracting the key points of the upper body skeleton in the RGB video and the key points of the lower body skeleton in the RGB video through the whole body posture estimator;

[0072] Step S120, obtaining the first position coordinates of the key points of the upper body skeleton;

[0073] Step S130, setting the human body key point of the lower body skeleton as the origin to obtain the second position coordinates.

[0074] Specifically, based on the COCO-WholeBody dataset, a full-body posture estimator is used to extract each human key point, and 133 human key points are obtained, and the 133 human key points are represented as coarse-grained features and fine-grained features, including 68 facial key points, 23 body key points, and 42 hand key points. Sign language is mainly based on hand movements, supplemented by complex expressions of the face and limbs, while coarse-grained features contain the overall structure and contextual information of the image, which is mainly manifested in the overall posture of the person and the obvious changes in limb movements, while fine-grained features provide more detailed local information, which is mainly reflected in the subtle changes in the hand joints and facial key points that occupy a smaller area in the image.

[0075] The COCO-WholeBody dataset is the first dataset with 133 human body key points, including 68 key points of the face, 17 key points of the body, 6 key points of the feet, and 21 key points of the left and right hands. The whole body pose estimator is the pose estimator HRNet pre-trained with whole body annotations. When the whole body pose estimator is used directly to extract 133 human body key points, the following is obtained: Figure 3 The key points of the human body are shown. Figure 3 The prediction of some human key points of the lower body skeleton is inaccurate. By correction, accurate prediction of human key points of the whole body can be obtained. Exemplarily, through the whole body posture estimator, the human key points of the whole body in the RGB video are extracted, the first position coordinates of the human key points of the upper body skeleton in the whole body human key points are obtained, the confidence of the human key points of the lower body skeleton is set to 0, and the second position coordinates of the human key points of the lower body skeleton are set to the origin (0,0), that is, the horizontal and vertical coordinates of the human key points of the lower body skeleton in the image display are both 0. After the above correction, it can be obtained Figure 4 The positions of the corrected key points of the human body are shown in the figure. There are 12 key points of the lower body skeleton, including 6 key points of the feet and the key points of the left hip, right hip, left knee, right knee, left ankle and right ankle on the body.

[0076] In step S200 of some embodiments, an initial heat map of the skeleton is generated based on the first position coordinates of the human body key points of the upper body skeleton and the second position coordinates of the human body key points of the lower body skeleton.

[0077] refer to Figure 5 In step S100 to step S200 shown in the embodiment of the present invention, an RGB video that can provide rich human appearance information is obtained, the RGB video is input into a pre-trained posture estimator, the position coordinates of 133 human key points are obtained, and the position coordinates are converted into a heat map. The heat map is used to represent the skeleton, which can make full use of all coarse-grained features and fine-grained features, and lay a good foundation for the subsequent more effective interaction between the skeleton modal features and the RGB video modal features.

[0078] like Figure 6As shown, in step S300 of some embodiments, the features of the RGB mode in the RGB video are linearly embedded through a linear layer to generate two feature vectors of the same size, namely, a first original feature and a first interactive feature. Exemplarily, the size of the first feature vector of the input RGB mode is B×T×(H×W)×C, where B is the batch size, T is the number of video frames, H is the frame height, W is the frame width, and C is the number of channels. Through a linear layer, the channel is mapped to a feature space of twice the dimension to obtain a second feature vector of size B×T×(H×W)×(2×C); through a ReLU activation function and a chunk function, the second feature vector is segmented according to a given channel dimension to obtain a first original feature and a first interactive feature, and the sizes of the first original feature and the first interactive feature are both B×T×(H×W)×c.

[0079] Before step S400 in some embodiments, the spatiotemporal cross-attention network sign language recognition method based on full-body features also includes modeling the initial heat map with the target position coordinates of each of the key points of the human body as the center to obtain a Gaussian heat map.

[0080] Specifically, there are 133 key points in the human skeleton structure. The initial heat map is modeled as a Gaussian distribution centered on the target position coordinates of each key point to obtain a Gaussian heat map. These Gaussian heat maps represent the existence of each corresponding key point. Since the horizontal and vertical coordinates of the key points of the lower body skeleton are both 0, the confidence is 0. Therefore, Figure 4 As shown in the figure, the human key points of the lower body skeleton are not displayed in the generated heat map. Storing the Gaussian heat map as a triplet of coordinates (x, y, c) is helpful to save storage space and minimize performance. c represents the maximum confidence of the Gaussian heat map, and (x, y) represents the position coordinates corresponding to the maximum confidence of the Gaussian heat map. For example, the position coordinates of the i-th human key point are (x i ,y i ), and the confidence of the position coordinate is z i , then the expression of the remaining pixels in the Gaussian heat map of the human key point is:

[0081]

[0082] Where Heatmap(x,y) represents the pixel of the Gaussian heat map of the key points of the human body; e represents the base of the natural logarithm; σ represents the radius of the Gaussian function.

[0083] like Figure 6As shown, in step S400 of some embodiments, the features of the skeleton mode in the initial heat map are linearly embedded through a linear layer to generate two feature vectors of the same size, namely, the second original feature and the second interactive feature. Exemplarily, the size of the third feature vector of the input skeleton mode is B×T×(H×W)×C, where B is the batch size, T is the number of video frames, H is the frame height, W is the frame width, and C is the number of channels. Through a linear layer, the channel is mapped to a feature space of twice the dimension to obtain a fourth feature vector of size B×T×(H×W)×(2×C); through a ReLU activation function and a chunk function, the fourth feature vector is segmented according to a given channel dimension to obtain a second original feature and a second interactive feature, and the sizes of the second original feature and the second interactive feature are both B×T×(H×W)×C.

[0084] Among them, the first original feature, the first interactive feature, the second original feature and the second interactive feature all contain coarse-grained features and fine-grained features. The coarse-grained features contain the overall structure and contextual information of the image, which are mainly manifested in the overall posture of the person and the obvious changes in body movements. The fine-grained features provide more detailed local information, which is mainly reflected in the subtle changes in the hand joints and facial key points that occupy a smaller area in the image. Considering coarse-grained features and fine-grained features can help to obtain features that can clearly distinguish sign language in the future.

[0085] In some embodiments, step S500 may include but is not limited to steps S510 to S580:

[0086] Step S510, generating a first query vector, a first key vector and a first value vector according to the first interaction feature;

[0087] Step S520, obtaining a global space template graph according to the first key vector and the first value vector;

[0088] Step S530, generating a second query vector, a second key vector and a second value vector according to the second interaction feature;

[0089] Step S540, obtaining a global context template graph according to the second key vector and the second value vector;

[0090] Step S550, obtaining a first feature map according to the global space template map and the second query vector;

[0091] Step S560, obtaining a second feature map according to the global context template map and the first query vector;

[0092] Step S570, aggregating the first feature map and the first original feature to obtain the first optimized feature;

[0093] Step S580: Aggregate the second feature map with the second original feature to obtain the second optimized feature.

[0094] In some embodiments, step S550 may include but is not limited to steps S551 to S553:

[0095] Step S551, presetting the zoom factor;

[0096] Step S552, normalizing the global space template image according to the scaling factor and the softmax function;

[0097] Step S553: ​​Map the normalized global space template map to the second query vector to obtain the first feature map.

[0098] In some embodiments, step S560 may include but is not limited to steps S561 to S563:

[0099] Step S561, presetting the zoom factor;

[0100] Step S562, normalizing the global context template graph according to the scaling factor and the softmax function;

[0101] Step S563: Map the normalized global context template graph to the first query vector to obtain the second feature graph.

[0102] In step S500 shown in the embodiment of the present invention, an improved spatiotemporal cross attention module is constructed to explore the intrinsic relationship between the RGB video and the skeleton in the feature extraction stage and to perform information interaction to improve the utilization rate of the features.

[0103] Specifically, the cross attention module is used to process the relationship between two sequences in natural language processing tasks. In sequence A, there is a query vector Q A , there is a key vector K in sequence B B , by querying the vector Q A With the key vector K B Perform operations such as dot multiplication to obtain the attention weight matrix, and apply the attention weight matrix to the value vector V in sequence B B To generate output, the expression of the initial attention weight matrix is:

[0104]

[0105] In the formula, m, n represent the corresponding row vectors; N represents the corresponding column vector; (·) T Represents a transpose operation; represents the scaling factor; Attention represents the output of the attention mechanism. The scaling factor can prevent the output of the attention mechanism from being too large or too small, and has an impact on the query vector Q A With the key vector K B The inner product of is scaled to avoid extreme values ​​in the output of the attention mechanism after normalization by the softmax function, which may cause problems such as gradient disappearance or gradient explosion, affecting the training efficiency and stability of the model.

[0106] In the query vector Q A and the key vector K B The computational complexity of the dot multiplication operation is as high as m×N 2 , and with the value vector V B After multiplication, the total amount of calculation reaches (m+n)×N 2 , ignoring the softmax function and scaling factor, the initial cross attention mechanism O is expressed as:

[0107] O=V B (K B T Q A )

[0108] The query vector, key vector, and value vector are rearranged to reduce the amount of calculation when the calculation results are equal. B With the key vector K B The computational complexity of the dot multiplication operation between them is only N×n×m times, and then combined with the query vector Q A After multiplication, the total amount of computation is only 2×N×n×m. Without considering the softmax function and the scaling factor, the improved cross attention mechanism O ′ The expression is:

[0109] O′=(V B K B T )Q A

[0110] According to the improved cross attention mechanism O ′ The expression of , combined with the normalization of the softmax function, can be obtained as follows:

[0111]

[0112] For example, Figure 6As shown, according to the first interactive feature, a first query vector Q1, a first key vector K1 and a first value vector V1 are generated. A dot multiplication operation is performed on the first key vector K1 and the first value vector V1 to obtain a global space template image containing RGB rich appearance information. The expression of the global space template image S1 is:

[0113] S1=K1V1 T

[0114] According to the second interaction feature, a second query vector Q2, a second key vector K2 and a second value vector V2 are generated. The second key vector K2 and the second value vector V2 are subjected to a dot multiplication operation to obtain a global context template graph containing rich temporal information in the skeleton. The expression of the global context template graph S2 is:

[0115] S2=K2V2 T

[0116] According to the global space template map and the second query vector, a first feature map containing RGB appearance information is obtained. The first feature map V skeleton The expression is:

[0117]

[0118] According to the global context template map and the first query vector, a second feature map containing skeleton structure motion information is obtained. The second feature map V rgb The expression is:

[0119]

[0120] The first feature map is added and aggregated with the first original feature to obtain an RGB optimized feature, that is, a first optimized feature; the second feature map is added and aggregated with the second original feature to obtain a skeleton optimized feature, that is, a second optimized feature.

[0121] Steps S500 to S700 shown in the embodiment of the present invention are as follows: Figure 2 As shown in Figure 2, several improved spatiotemporal cross attention modules are inserted into the deep feature extraction stage of RGB and skeleton to promote the information interaction between RGB modality and skeleton modality. Figure 6The structure of the plug-and-play spatiotemporal cross attention module shown in the figure generates interactive features of the two modalities according to the RGB modality and the skeleton modality, and generates query vectors, key vectors and value vectors of the two modalities. Then, the global space template graph and the global context template graph are obtained by multiplying the key vector and the value vector in the attention mechanism. Then, the query vector of the skeleton modality is multiplied with the global space template graph, and the query vector of the RGB modality is multiplied with the global context template graph, and finally, the original features of the modality are added and aggregated to obtain the optimized features. Each optimized feature will enter the next deep feature extraction process of the three-dimensional residual network, and the improved spatiotemporal cross attention module is inserted in the deep extraction process to optimize the features until entering the last stage of the feature extraction process. After the RGB modality and the skeleton modality pass through the last improved spatiotemporal cross attention module, the final RGB optimized features and the final skeleton optimized features are obtained. The final RGB optimized features and the final skeleton optimized features are directly fused through the channel splicing operation to obtain the fused features, and then the fused features are passed through the classifier to obtain the sign language recognition results. Realizing the interaction of deep information of bimodal data in the feature extraction stage, exploring the intrinsic relationship between RGB and skeleton in the feature extraction stage, and conducting deep information interaction is conducive to absorbing the advantageous features of the other modality, enhancing the feature representation capabilities of the two modalities, and improving the feature utilization and overall performance.

[0122] like Figure 7 As shown, the embodiment of the present invention also provides a spatiotemporal cross-attention network sign language recognition device 800 based on full-body features, which can implement the above-mentioned spatiotemporal cross-attention network sign language recognition method based on full-body features, and the device includes:

[0123] The first module 801 is used to input the RGB video into a whole body posture estimator to obtain the target position coordinates of the key points of the human body;

[0124] The second module 802 is used to generate an initial heat map of the skeleton according to the target position coordinates;

[0125] The third module 803 is used to generate a first original feature and a first interactive feature according to the RGB video;

[0126] The fourth module 804 is used to generate a second original feature and a second interactive feature according to the initial heat map;

[0127] The fifth module 805 is used to input the first original feature, the first interactive feature, the second original feature and the second interactive feature into a spatiotemporal cross attention module for optimization processing to obtain a first optimized feature and a second optimized feature;

[0128] The sixth module 806 is used to perform a splicing operation on the first optimization feature and the second optimization feature to obtain a fusion feature;

[0129] The seventh module 807 is used to input the fusion feature into the classifier to obtain the sign language recognition result.

[0130] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0131] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned spatiotemporal cross-attention network sign language recognition method based on full-body features is implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0132] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] refer to Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0134] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0135] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the spatiotemporal cross-attention network sign language recognition method based on full-body features in the embodiments of this application;

[0136] Input / output interface 903, used to implement information input and output;

[0137] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0138] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0139] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0140] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned spatiotemporal cross-attention network sign language recognition method based on full-body features.

[0141] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the aforementioned spatiotemporal cross-attention network sign language recognition method based on full-body features.

[0143] In summary, the spatiotemporal cross-attention network sign language recognition method and device based on full-body features according to the embodiment of the present invention has the following advantages:

[0144] 1. The 133 key point features of the human body, namely, hands, face and limbs, used in the embodiment of the present invention can fully represent sign language. Considering the coarse-grained features from the limbs and the fine-grained features from the face and hands can help the subsequent network extract features that can more clearly distinguish sign language.

[0145] 2. The embodiment of the present invention uses a heat map to represent the skeleton of the whole body, which can fully utilize the coarse-grained features from the limbs and the fine-grained features from the face and hands, and improve the utilization rate of the 133 key point features of the whole body. In addition, representing the skeleton of the whole body in the form of a heat map can reduce the model complexity caused by too many key points of the whole body, and is conducive to the subsequent feature-level information interaction with the RGB modality in the feature extraction stage.

[0146] 3. The embodiment of the present invention improves the cross-attention mechanism and constructs a spatiotemporal cross-attention module. Through the plug-and-play structure of the spatiotemporal cross-attention module, it realizes the interaction of deep information between RGB video modality and skeleton modality data in the feature extraction stage, absorbs the advantageous features of the other modality, enhances the feature representation capabilities of the two modalities, and improves the feature utilization and overall performance.

[0147] 4. The embodiment of the present invention only uses RGB video modality and skeleton modality. Compared with methods with more modalities, the embodiment of the present invention improves the real-time performance of the online sign language system, reduces the complexity of the model, and also improves the accuracy of sign language recognition.

[0148] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0149] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0150] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0151] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0152] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0153] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0154] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0155] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0156] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A sign language recognition method based on spatiotemporal cross-attention network with full-body features, characterized in that: include: Input the RGB video into the full body posture estimator to obtain the target position coordinates of the key points of the human body; Generating an initial heat map of the skeleton according to the target position coordinates; Generate a first original feature and a first interactive feature according to the RGB video; Generate a second original feature and a second interactive feature according to the initial heat map; Inputting the first original feature, the first interactive feature, the second original feature, and the second interactive feature into a spatiotemporal cross attention module for optimization processing to obtain a first optimized feature and a second optimized feature; Performing a splicing operation on the first optimized feature and the second optimized feature to obtain a fusion feature; Inputting the fused features into a classifier to obtain a sign language recognition result; The step of inputting the RGB video into a full-body posture estimator to obtain the target position coordinates of key points of the human body includes: Extracting the key points of the upper body skeleton in the RGB video and the key points of the lower body skeleton in the RGB video by the whole body posture estimator; Obtaining first position coordinates of key points of the upper body skeleton; The key point of the human body of the lower body skeleton is set as the origin to obtain the second position coordinate.

2. According to the method of claim 1, the method is characterized in that: Before generating the second original feature and the second interactive feature, the method further includes: The initial heat map is modeled with the target position coordinates of each of the key points of the human body as the center to obtain a Gaussian heat map.

3. The method for sign language recognition based on spatiotemporal cross-attention network based on full-body features according to claim 1, characterized in that: The first original feature, the first interactive feature, the second original feature and the second interactive feature are input into the spatiotemporal cross attention module for optimization processing to obtain a first optimized feature and a second optimized feature, include: generating a first query vector, a first key vector, and a first value vector according to the first interaction feature; Obtaining a global space template graph according to the first key vector and the first value vector; generating a second query vector, a second key vector, and a second value vector according to the second interaction feature; Obtaining a global context template graph according to the second key vector and the second value vector; Obtaining a first feature map according to the global space template map and the second query vector; Obtaining a second feature graph according to the global context template graph and the first query vector; Aggregating the first feature map with the first original feature to obtain the first optimized feature; The second feature map is aggregated with the second original feature to obtain the second optimized feature.

4. The method for sign language recognition based on spatiotemporal cross-attention network based on full-body features according to claim 3, characterized in that: The obtaining a first feature map according to the global space template map and the second query vector includes: Preset zoom factor; Normalizing the global space template map according to the scaling factor and the softmax function; The normalized global space template map is mapped to the second query vector to obtain the first feature map.

5. The method for sign language recognition based on spatiotemporal cross-attention network based on full-body features according to claim 3, characterized in that: The obtaining a second feature graph according to the global context template graph and the first query vector includes: Preset zoom factor; Normalizing the global context template graph according to the scaling factor and the softmax function; The normalized global context template map is mapped to the first query vector to obtain the second feature map.

6. A spatiotemporal cross-attention network sign language recognition device based on full-body features, characterized in that: include: The first module is used to input the RGB video into the whole body posture estimator to obtain the target position coordinates of the key points of the human body; The second module is used to generate an initial heat map of the skeleton according to the target position coordinates; A third module is used to generate a first original feature and a first interactive feature according to the RGB video; A fourth module is used to generate a second original feature and a second interactive feature according to the initial heat map; A fifth module, configured to input the first original feature, the first interactive feature, the second original feature, and the second interactive feature into a spatiotemporal cross attention module for optimization processing to obtain a first optimized feature and a second optimized feature; A sixth module is used to perform a splicing operation on the first optimization feature and the second optimization feature to obtain a fusion feature; The seventh module is used to input the fusion feature into a classifier to obtain a sign language recognition result; Wherein, the first module is specifically used for: Extracting the key points of the upper body skeleton in the RGB video and the key points of the lower body skeleton in the RGB video by the whole body posture estimator; Obtaining first position coordinates of key points of the upper body skeleton; The key point of the human body of the lower body skeleton is set as the origin to obtain the second position coordinate.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images