Sign language recognition method and device, storage medium and electronic device

Abstract

The present disclosure provides a sign language recognition method, device, storage medium and electronic device, which at least to some extent overcome the problem of inaccurate sign language recognition in related technologies. The sign language recognition method includes: detecting the whole body key points of the human body in the image to be detected to obtain the whole body key point data of the human body; extracting the target key point data related to the sign language from the whole body key point data of the human body; constructing a spatiotemporal graph structure using the target key point data; identifying the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to the spatiotemporal graph convolutional neural network model obtained by pre-training, wherein in the process of training the spatiotemporal graph convolutional neural network model, the features of some key points are removed each time training, and the embodiment of the present disclosure improves the accuracy of sign language recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and in particular to a sign language recognition method, device, storage medium and electronic device. Background Art

[0002] At present, smart home robots can provide auxiliary functions for voice commands, but for some deaf-mute people or elderly people with language barriers, it is impossible to give commands to the robot through voice. In this case, the sign language command function can be turned on, and the user's sign language video can be obtained through the camera and the corresponding commands can be recognized, so as to achieve the same function as voice. This can expand the user base of smart home robots. At present, sign language recognition is still a huge challenge. The reason is that sign language recognition is different from ordinary gesture recognition. Sign language is faster and more complex, and some body and even facial movements are added to express a complete meaning. Recently, some solutions have performed sign language recognition based on skeletal motion, but this recognition method relies on skeletal motion annotation of motion capture systems such as Kinect, which makes it limited to specific data sets or specific environments and performs poorly in non-specific environments. Secondly, this recognition only focuses on the movement trajectory of the human skeleton, resulting in incomplete and inaccurate expression information. Summary of the invention

[0003] The present disclosure provides a sign language recognition method, device, storage medium and electronic device, which at least to some extent overcome the problem of inaccurate sign language recognition in related technologies.

[0004] According to a first aspect of the present disclosure, a sign language recognition method is provided, comprising: detecting whole-body key points of a human body in an image to be detected to obtain whole-body key point data of the human body; extracting target key point data related to sign language from the whole-body key point data of the human body; constructing a spatiotemporal graph structure using the target key point data; and identifying the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to a pre-trained spatiotemporal graph convolutional neural network model, wherein during the training of the spatiotemporal graph convolutional neural network model, the features of some key points are removed each time the training is performed.

[0005] Optionally, the method also includes: before identifying the type of sign language contained in the image to be detected based on the space-time graph structure according to the pre-trained space-time graph convolutional neural network model, training the space-time graph convolutional neural network model includes multiple training stages that are repeatedly executed, and each training stage includes: performing spatial graph convolution on the features of the nodes on the first space-time graph constructed based on the training data to obtain a second space-time graph; reallocating weights on the features of the nodes on the second space-time graph in at least three different dimensions to obtain a third space-time graph; performing temporal graph convolution on the features of the nodes on the third space-time graph to obtain a fourth space-time graph; removing the features of some nodes on the fourth space-time graph, and superimposing the remaining features with the features obtained in the previous training stage of the current training stage to obtain the features output by the current training stage.

[0006] Optionally, the features of some key points removed during each training include: features of any two adjacent key points that are randomly removed.

[0007] Optionally, the weights of the features of the nodes on the second space-time graph are reallocated in at least three different dimensions to obtain a third space-time graph, including: inputting the features of the nodes of the second space-time graph into a spatial self-attention model, and assigning a first weight to the features of all nodes belonging to the same frame; multiplying the first weight assigned to each feature by the feature itself and then adding the weights to the features to obtain a first feature vector; inputting the first feature vector into a temporal self-attention model, and assigning a second weight to the features of the same node belonging to different frames; multiplying the second weight assigned to each feature by the feature itself and then adding the weights to the features to obtain a second feature vector; inputting the second feature vector into a channel self-attention model, and assigning a third weight to the features of the same node in different channels; multiplying the third weight assigned to each feature by the feature itself and then adding the weights to the features to obtain a third feature vector.

[0008] Optionally, the target key point data includes: skeletal features, motion features, joint motion features and skeletal motion features, and the type of sign language contained in the image to be detected is identified based on the spatiotemporal graph structure according to a pre-trained spatiotemporal graph convolutional neural network model, including: inputting the skeletal features, the motion features, the joint motion features and the skeletal motion features into the spatiotemporal graph convolutional neural network model to obtain joint feature prediction results, skeletal feature prediction results, joint motion feature prediction results and skeletal motion feature prediction results output by the spatiotemporal graph convolutional neural network model; and fusing the joint feature prediction results, the skeletal feature prediction results, the joint motion feature prediction results and the skeletal motion feature prediction results to obtain the type of sign language contained in the image to be detected.

[0009] Optionally, the method also includes: after identifying the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to the pre-trained spatiotemporal graph convolutional neural network model, displaying an animated image for displaying the type of sign language according to the identified type of sign language.

[0010] Optionally, the target key point data includes data of human finger joint points.

[0011] According to a second aspect of the present disclosure, a sign language recognition device is provided, comprising: a detection module, used to detect whole-body key points of a human body in an image to be detected to obtain whole-body key point data of the human body; an extraction module, used to extract target key point data related to sign language from the whole-body key point data of the human body; a construction module, used to construct a spatiotemporal graph structure using the target key point data; a classification module, used to identify the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to a pre-trained spatiotemporal graph convolutional neural network model, wherein in the process of training the spatiotemporal graph convolutional neural network model, the features of some key points are removed each time training.

[0012] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the sign language recognition methods provided by the embodiments of the present disclosure by executing the executable instructions.

[0013] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the sign language recognition methods provided in the embodiments of the present disclosure is implemented.

[0014] The sign language recognition method, device, storage medium and electronic device of the embodiments of the present disclosure detect the whole body key points of the human body in the image to be detected during sign language recognition, and then extract the target key points related to the sign language from the detected whole body key point data, so that the extracted target key points can fully express the characteristic movements of the sign language, and then use the target key point data to construct a spatiotemporal graph structure, and use the pre-trained spatiotemporal graph convolutional neural network model to identify the type of sign language contained in the image to be detected based on the spatiotemporal graph structure, thereby improving the accuracy of sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a sign language recognition method according to one or more embodiments of the present disclosure;

[0016] Figure 2is a schematic diagram of a space-time graph structure according to one or more embodiments of the present disclosure;

[0017] Figure 3 is a flowchart of a training phase of a spatiotemporal graph convolutional neural network model according to one or more embodiments of the present disclosure;

[0018] Figure 4 is a training process of a spatiotemporal graph convolutional neural network according to one or more embodiments of the present disclosure;

[0019] Figure 5 is a flowchart of reallocating weights of features of nodes on the second spatiotemporal graph in at least three different dimensions according to one or more embodiments of the present disclosure;

[0020] Figure 6 is a flowchart of reallocating weights of features of nodes on a second spatiotemporal graph in three dimensions of time, space, and channel according to one or more embodiments of the present disclosure;

[0021] Figure 7 is a flowchart of identifying the type of sign language contained in an image to be detected according to a pre-trained spatiotemporal graph convolutional neural network model according to one or more embodiments of the present disclosure;

[0022] Figure 8 is a schematic diagram of training a spatiotemporal graph convolutional neural network model with four branches according to one or more embodiments of the present disclosure;

[0023] Fig. 9 is a flow chart of a sign language recognition method according to one or more embodiments of the present disclosure;

[0024] Fig.10 is a structural schematic diagram of a sign language recognition device according to one or more embodiments of the present disclosure;

[0025] and

[0026] Fig.11 It is a schematic diagram of the structure of an electronic device according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] Figure 1 is a flowchart of a sign language recognition method according to one or more embodiments of the present disclosure. Figure 1 As shown, the method includes:

[0030] Step S102: detecting the key points of the whole body of the human body in the image to be detected, and obtaining the key points data of the whole body of the human body;

[0031] Optionally, before step S102, a video or image of the sign language presenter may be acquired through a camera to obtain the above-mentioned image to be detected.

[0032] Optionally, a full-body key point detection network with a pre-trained model can be used to extract full-body key points of the human body based on the image to be detected.

[0033] Step S104: extracting target key point data related to the gesture from the key point data of the human body;

[0034] The inventors of the present disclosure have discovered that human motion prediction only requires a dozen key points, while the above-mentioned extraction results in as many as 133 key points of the human body. Too many key points will bring noise to the training model, and if two key points are too far apart, and if there are many key points between them, the motion relationship between the two key points will be difficult to learn, which will reduce the final recognition accuracy. Therefore, the inventors of the present disclosure, through observation of the data and the working principle of the graph convolutional network, simplified the whole body key points after many experiments, and reduced the 133 key points to 27 main key points related to sign language, including 20 key points of the fingers of both hands and 7 key points of other parts of the human body (for example, arms, neck and head). The simplified spatiotemporal graph structure can be shown as follows: Figure 2 As shown in the figure, the solid dots represent the spatial graph structure formed by connecting all the key points on a frame of image according to the structure of the human body, and the hollow dots represent the time graph structure formed by connecting the key points of the same part between frames. The simplified key points can not only fully express the specific movements of sign language, but also speed up the calculation speed of the model and improve the recognition accuracy.

[0035] Step S106: constructing a spatiotemporal graph structure using the target key point data;

[0036] Optionally, in space, all key points on a frame of image can be connected according to the structure of the human body to form a spatial graph structure, and key points of the same part can be connected between frames to form a temporal graph structure. For example, the constructed spatiotemporal graph structure can be as follows: Figure 2 shown.

[0037] Step S108: Identify the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to the pre-trained spatiotemporal graph convolutional neural network model, wherein during the training of the spatiotemporal graph convolutional neural network model, features of some key points are removed each time during training.

[0038] Optionally, the features of all nodes of the spatiotemporal graph structure may be input into the spatiotemporal graph convolutional neural network model, and the output of the spatiotemporal graph convolutional neural network model represents the type of sign language contained in the image to be detected.

[0039] Optionally, the spatiotemporal graph convolutional neural network model may require repeated training in multiple training stages. After obtaining the output features in each training stage, part of the output features may be removed to reduce the amount of calculation and accelerate the convergence of the network.

[0040] The sign language recognition method of the disclosed embodiment detects the whole body key points of the human body in the image to be detected during sign language recognition, and then extracts the target key points related to the sign language from the detected whole body key point data, so that the extracted target key points can fully express the characteristic movements of the sign language, and then uses the target key point data to construct a spatiotemporal graph structure, and uses a pre-trained spatiotemporal graph convolutional neural network model to identify the type of sign language contained in the image to be detected based on the spatiotemporal graph structure, thereby improving the accuracy of sign language recognition.

[0041] In addition, during the repeated training of the spatiotemporal graph convolutional neural network model, the features of some key points are removed each time, which can reduce the amount of calculation and accelerate the convergence of the model.

[0042] The spatiotemporal graph convolutional neural network model of the disclosed embodiment can be deployed on an intelligent robot. For some users who cannot issue commands to the robot through voice, the sign language command function can be turned on, and the user's sign language video can be obtained through the camera and the corresponding command can be recognized, thereby achieving the same function as issuing voice commands. This expands the user group of home robots.

[0043] In one or more embodiments of the present disclosure, the sign language recognition method may further include:

[0044] Before the pre-trained spatiotemporal graph convolutional neural network model identifies the type of sign language contained in the image to be detected based on the spatiotemporal graph structure, the spatiotemporal graph convolutional neural network model is trained, wherein the training of the spatiotemporal graph convolutional neural network model includes a plurality of training stages that are repeatedly performed, such as Figure 3 As shown, each training phase may include:

[0045] Step S302: performing spatial graph convolution on the features of the nodes on the first spatiotemporal graph constructed based on the training data to obtain a second spatiotemporal graph;

[0046] Step S304: reallocating weights to the features of the nodes on the second space-time graph in at least three different dimensions to obtain a third space-time graph;

[0047] The at least three different dimensions may include, for example, a time dimension, a space dimension, and a channel dimension.

[0048] Step S306: performing time graph convolution on the features of the nodes on the third space-time graph to obtain a fourth space-time graph;

[0049] Step S308: removing features of some nodes on the fourth spatiotemporal graph, and superimposing the remaining features with features obtained in a previous training stage of the current training stage to obtain features outputted in the current training stage.

[0050] Figure 4 The following is an example of a training process of a spatiotemporal graph convolutional neural network model. Figure 4 As shown, the input passes through the spatial convolution layer 410 and then enters the BN (Batch Normalization) layer 412 to prevent overfitting and accelerate convergence. Then it enters the self-attention module 416 through the Relu activation layer 414, which can redistribute the weights of the nodes on the spatiotemporal graph in the three dimensions of space, time and channel, and then connects to the BN layer 418 and the Relu activation layer 420 to enter the temporal convolution layer 422, and also connects to the BN layer 424. Finally, the graph reduction 426 operation is performed to discard the features of adjacent nodes. The input of the previous training stage is also subjected to the graph reduction 4100 and the output is added and output through the Relu activation layer 4102.

[0051] Figure 4 The process shown is the working process of the entire sign language recognition graph convolution module. In the embodiment of the present disclosure, multiple, for example, 10 such modules can be connected in series to form the final spatiotemporal graph convolution neural network. Finally, softmax is used to classify the sign language movements to identify the meaning of the sign language.

[0052] The following is a brief description of the spatial graph convolution and temporal graph convolution involved in the embodiments of the present disclosure.

[0053] like Figure 2 In the space-time graph structure shown in the figure, the key point set V is defined as {v i,t |i=1,....,N,t=1,.....,T}, where N=27 is all the key points in a frame, and T is the total number of frames of the image to be detected. The adjacency matrix A is defined as shown in formula (1):

[0054]

[0055] Among them, d is calculated as v i and v j The shortest distance is the distance from v i Walk along the edge through the least critical points to reach v j , if v i and v j If connected, d = 1. The spatial graph convolution can be expressed by the following formula (2):

[0056] X out =D -1 / 2 (A+I)D -1 / 2 X in W (2)

[0057] Where A represents the adjacency matrix as described above, I represents the identity matrix, D represents the diagonal matrix of A+I, and W is the parameter matrix to be trained. in is the input feature, where X in It can be expressed as {x, y, score}, where x, y, and score represent the horizontal and vertical coordinates and the key point confidence, respectively. These data can be obtained by the key point detection network, which is also pre-trained. out To output features, the entire formula represents a spatial graph convolution operation. After continuous spatial graph convolution operations, the parameters in W can be continuously updated to train the network.

[0058] In the time dimension, assuming that the time interval is 5 frames, all key points of the same part in the 5 frames can be connected to form a graph structure in the time dimension, such as Figure 2 The hollow dots are connected by dotted lines. The graph structure in the time dimension is divided according to the number of key points in a single frame. For example, if there are 27 key points in total, 27 graph structures will be formed in the time dimension. Temporal graph convolution is to perform the above-mentioned operations similar to spatial convolution on these 27 graph structures.

[0059] Spatial convolution and temporal convolution can often be performed alternately, that is, the features can be continuously convolved in space and time, and the final W is the result of the joint training of the two.

[0060] In one or more embodiments of the present disclosure, removing features of some key points during each training may include: randomly removing features of any two adjacent key points. Optionally, the operation of randomly removing features of any two adjacent key points may be performed by Figure 4 The graph reduction module shown in is executed. Randomly discarding the features of any two adjacent key points can reduce the amount of calculation and speed up the convergence of the network. On the other hand, since the action changes of two adjacent key points are not rich for the human body, and the more key points in the middle, the richer the action combinations, so discarding some features not only does not affect the training results, but also has the effect of improving the recognition accuracy of the model.

[0061] like Figure 5 As shown, reallocating weights to the features of the nodes on the second spatiotemporal graph in at least three different dimensions to obtain a third spatiotemporal graph may include:

[0062] Step S3042: inputting the features of the second spatiotemporal graph nodes into the spatial self-attention model, and assigning a first weight to the features of all nodes belonging to the same frame;

[0063] S3044: multiplying the first weight assigned to each feature by each feature itself and then adding the weight to each feature itself to obtain a first feature vector;

[0064] Optional, such as Figure 6 As shown, the features of all nodes in the second spatiotemporal graph are input into the spatial self-attention model. After the spatial self-attention model, weights are assigned to all nodes in the same frame. In order to distinguish them from the weights assigned in the subsequent process, the weights are recorded as the first weights. The first weights assigned to each feature are multiplied by each feature itself and then added to each feature itself to obtain the first feature vector, which can be expressed as f'=w*f+f, where f represents the feature vector of a node, w represents the first weight assigned after the spatial self-attention model, and f' represents the output feature vector, that is, the first feature vector.

[0065] S3046: Input the first feature vector into the temporal self-attention model, and assign a second weight to features of the same node belonging to different frames;

[0066] S3048: Multiply the second weight assigned to each feature by the feature itself and then add the weight to the feature itself to obtain a second feature vector;

[0067] Optional, still with Figure 6As shown, the temporal self-attention model assigns weights to the same node features between different frames, multiplies the second weight assigned to each feature by the feature itself, and then adds the weight to the feature itself to obtain the second feature vector, which can also be expressed as f'=w*f+f, where f represents the feature vector of a node, w represents the second weight assigned after the temporal self-attention model, and f' represents the output feature vector, that is, the second feature vector.

[0068] S30410: inputting the second feature vector into a channel self-attention model, and assigning a third weight to features of the same node in different channels;

[0069] S30412: Multiply the third weight assigned to each feature by each feature itself and then add the weights to each feature itself to obtain a third feature vector.

[0070] Optional, still with Figure 6 As shown, the channel self-attention model assigns weights to features of the same node but in different channels. The third weight assigned to each feature is multiplied by each feature itself and then added to each feature itself, which can be expressed as f'=w*f+f, where f represents the feature vector of a node, w represents the third weight assigned after the channel self-attention model, and f' represents the output feature vector, that is, the third feature vector in the image convolution, the number of convolution kernels in a convolution layer will get the number of feature maps, and the number of feature maps is the number of channels. Similarly, in spatiotemporal graph convolution, for a certain node, multiple convolution kernels can be used to extract its features, that is, a node can get multiple feature vectors, and the number of feature vectors is the number of channels.

[0071] In one or more embodiments of the present disclosure, the target key point data may include: bone features, action features, joint point motion features and bone motion features, such as Figure 7 As shown, identifying the type of sign language contained in the image to be detected based on the space-time graph structure according to the pre-trained space-time graph convolutional neural network model may include:

[0072] Step S702: inputting the bone features, the action features, the joint point movement features and the bone movement features into the spatiotemporal graph convolutional neural network model to obtain joint feature prediction results, bone feature prediction results, joint point movement feature prediction results and bone movement feature prediction results output by the spatiotemporal graph convolutional neural network model;

[0073] Optionally, the expression of sign language can be expressed by human joints, bones, joint movements and bone movements. Based on this, it can be divided into four branches to extract features from the above four aspects respectively. Figure 7The figure shows the structure of the multi-branch spatiotemporal graph convolutional network model. The skeleton data can be obtained from the joint features at the beginning of the action and the joint features after the action ends, such as Figure 2 The central node shown represents the nose of a person. It is taken as the root node and its bone feature is 0. The initial joint point feature and the target joint point feature are defined as and x i,t ,y i.t is the coordinate representation of the joint starting point, x j,t ,y j.t is the coordinate of the end point of the joint point. Where x, y, s are the xy coordinates and confidence of the joint point respectively. The skeleton feature can be obtained by subtracting the initial joint feature from the target joint feature, that is, the skeleton feature can be expressed as The action feature can be calculated by the difference between the joint point features and the skeleton features of two adjacent frames; the joint point action feature is calculated by the difference between the joint point features of two adjacent frames, that is, the joint point action feature can be expressed as The skeletal motion feature can be calculated by the difference between the skeletal features of two adjacent frames. For example, the skeletal motion feature can be expressed as

[0074] Step S704: Fusing the joint feature prediction results, the bone feature prediction results, the joint point motion feature prediction results and the bone motion feature prediction results to obtain the type of sign language contained in the image to be detected.

[0075] Optional, such as Figure 8 As shown, during model training, the above four branches can be trained separately, and finally the parameters obtained from the training can be fused. The fusion method is, for example, to perform weighted summation on the parameters obtained from the training, wherein the weights of the four branches can be set by experiments and actual conditions. Based on this, the trained model can predict the type of sign language based on skeletal feature data, motion feature data, joint motion feature data, and skeletal motion feature data, and the weighted summation of the obtained prediction results can be performed to obtain the final prediction result, which can make the final prediction result more accurate.

[0076] Fig. 9 is a flowchart of a sign language recognition method according to one or more embodiments of the present disclosure. Fig. 9 As shown, this method Figure 1 Based on the method shown, it may also include:

[0077] Step S902: After the pre-trained spatiotemporal graph convolutional neural network model identifies the type of sign language contained in the image to be detected based on the spatiotemporal graph structure, an animated image for displaying the type of sign language is displayed according to the identified type of sign language.

[0078] Optionally, the type of sign language identified may include the meaning of the sign language, and an animation for displaying the sign language may be generated according to the type of sign language identified or the meaning of the sign language; or an animation corresponding to the type or meaning of the sign language that is pre-saved may be obtained according to the type of sign language identified or the meaning of the sign language. For example, an animation corresponding to the type of sign language or the meaning of the sign language may be displayed by a smart home robot. So that the user can determine whether the sign language he made is correctly recognized through the animation, which facilitates information interaction between users with language barriers and smart home robots. For example, for deaf-mute elderly people, they cannot issue instructions to the smart home robot through voice, and they cannot receive voice feedback from the smart home robot. Based on the sign language recognition method of one or more embodiments of the present disclosure, it is only necessary to make a sign language action in front of the camera of the smart home robot. After the smart home robot recognizes it, it will display an animated character on the screen and give corresponding feedback by displaying the sign language action through the animated character. This facilitates the use of smart home robots by deaf-mute elderly people.

[0079] In one or more embodiments of the present disclosure, the target key point data includes data of human finger joints. Optionally, the target key point data may include data of each joint point of the fingers of both hands of the human body. The movement trajectory of the finger is a main feature of sign language and plays a vital role in the recognition of sign language. Therefore, the sign language recognition method of one or more embodiments of the present disclosure takes into account the characteristics of the finger joints when performing sign language recognition. The characteristics may include finger bone characteristics, finger action characteristics, finger joint point characteristics and finger bone action characteristics, as well as finger joint point action characteristics. The calculation method of these features can refer to the calculation method of the bone characteristics, joint characteristics, joint action characteristics and bone action characteristics recorded above, which will not be repeated here. Correspondingly, the training data when training the above-mentioned spatiotemporal graph convolutional network model also includes human finger joint data, so that the model pays attention to the movement trajectory of the fingers, which can make the expression of information more complete and accurate, thereby also improving the accuracy of sign language recognition.

[0080] Fig.10 is a structural diagram of a sign language recognition device according to one or more embodiments of the present disclosure. Fig.10 As shown, the device 1010 includes:

[0081] A detection module 1012 is used to detect the key points of the whole body of the human body in the image to be detected, and obtain the key points data of the whole body of the human body;

[0082] An extraction module 1014 is used to extract target key point data related to sign language from the key point data of the whole body of the human body;

[0083] A construction module 1016 is used to construct a spatiotemporal graph structure using the target key point data;

[0084] The classification module 1018 is used to identify the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to the pre-trained spatiotemporal graph convolutional neural network model, wherein during the training process of the spatiotemporal graph convolutional neural network model, the features of some key points are removed each time the training is performed.

[0085] In one or more embodiments of the present disclosure, the above-mentioned sign language recognition device may further include:

[0086] A training module is used to train the spatiotemporal graph convolutional neural network model before identifying the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to the pre-trained spatiotemporal graph convolutional neural network model, wherein the training of the spatiotemporal graph convolutional neural network model includes a plurality of training stages that are repeatedly executed, and each training stage includes:

[0087] Performing spatial graph convolution on the features of nodes on the first spatiotemporal graph constructed based on the training data to obtain a second spatiotemporal graph;

[0088] Reassign weights to features of nodes on the second spatiotemporal graph in at least three different dimensions to obtain a third spatiotemporal graph;

[0089] Performing time graph convolution on the features of the nodes on the third space-time graph to obtain a fourth space-time graph;

[0090] The features of some nodes on the fourth spatiotemporal graph are removed, and the remaining features are superimposed with the features obtained in the previous training stage of the current training stage to obtain the features outputted in the current training stage.

[0091] In one or more embodiments of the present disclosure, the features of some key points removed during each training include:

[0092] The features of any two adjacent key points are randomly removed.

[0093] In one or more embodiments of the present disclosure, the training module may be used to:

[0094] Inputting the features of the nodes of the second spatiotemporal graph into a spatial attention model, and assigning a first weight to the features of all nodes belonging to the same frame;

[0095] The first weight assigned to each feature is multiplied by each feature itself and then added to each feature itself to obtain a first feature vector;

[0096] Inputting the first feature vector into a temporal attention model, and assigning a second weight to features of the same node belonging to different frames;

[0097] The second weight assigned to each feature is multiplied by each feature itself and then added to each feature itself to obtain a second feature vector;

[0098] Inputting the second feature vector into a channel attention model, and assigning a third weight to features of the same node in different channels;

[0099] The third weight assigned to each feature is multiplied by each feature itself and then added to each feature itself to obtain a third feature vector.

[0100] In one or more embodiments of the present disclosure, the target key point data may include: skeleton feature data, motion feature data, joint point motion features and skeleton motion features, and the classification module may be specifically used for:

[0101] Inputting the skeleton feature data, the motion feature data, the joint motion feature data and the skeleton motion feature data into the spatiotemporal graph convolutional neural network model, and obtaining joint feature prediction results, skeleton feature prediction results, joint motion feature prediction results and skeleton motion feature prediction results output by the spatiotemporal graph convolutional neural network model;

[0102] The joint feature prediction results, the bone feature prediction results, the joint point motion feature prediction results and the bone motion feature prediction results are fused to obtain the type of sign language contained in the image to be detected.

[0103] In one or more embodiments of the present disclosure, the sign language recognition device may further include:

[0104] A display module is used to display an animation of the sign language of the type identified according to the type of sign language after the type of sign language contained in the image to be detected is identified based on the space-time graph structure according to the pre-trained space-time graph convolutional neural network model.

[0105] In one or more embodiments of the present disclosure, the target key point data includes data of human finger joint points.

[0106] One or more embodiments of the present disclosure further provide an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the sign language recognition methods provided by the embodiments of the present disclosure by executing the executable instructions.

[0107] One or more embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, any one of the sign language recognition methods provided by the embodiments of the present disclosure is implemented.

[0108] Refer to the following Fig.11 The electronic device 1100 according to this embodiment of the present invention is described. Fig.11 The electronic device 1100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0109] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include but are not limited to: at least one processing unit 1110, at least one storage unit 1120, and a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110).

[0110] The storage unit stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1110 can perform the following steps: Figure 1 Step S102: detect the whole body key points of the human body in the image to be detected to obtain the whole body key point data of the human body; step S104: extract the target key point data related to the gesture from the whole body key point data of the human body; step S106: construct a spatiotemporal graph structure using the target key point data; identify the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to the pre-trained spatiotemporal graph convolutional neural network model, wherein in the process of training the spatiotemporal graph convolutional neural network model, the features of some key points are removed each time training.

[0111] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 11201 and / or a cache storage unit 11202 , and may further include a read-only storage unit (ROM) 11203 .

[0112] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0113] Bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0114] The electronic device 1100 may also communicate with one or more external devices 1200 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0115] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0116] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0117] A program product for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0118] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0119] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0120] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0121] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0122] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0123] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0124] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0125] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A sign language recognition method, characterized in that: include: Detect the key points of the human body in the image to be detected to obtain the key points data of the human body; Extracting target key point data related to sign language from the human body key point data; Constructing a spatiotemporal graph structure using the target key point data; Identify the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to the pre-trained spatiotemporal graph convolutional neural network model, wherein during the training of the spatiotemporal graph convolutional neural network model, the features of some key points are removed in each training; Wherein, before identifying the type of sign language contained in the image to be detected based on the space-time graph structure according to the pre-trained space-time graph convolutional neural network model, the method also includes: training the space-time graph convolutional neural network model, wherein the training of the space-time graph convolutional neural network model includes multiple training stages that are repeatedly executed, and each training stage includes: performing spatial graph convolution on the features of the nodes on the first space-time graph constructed based on the training data to obtain a second space-time graph; reallocating weights on the features of the nodes on the second space-time graph in at least three different dimensions to obtain a third space-time graph; performing temporal graph convolution on the features of the nodes on the third space-time graph to obtain a fourth space-time graph; removing the features of some nodes on the fourth space-time graph, and superimposing the remaining features with the features obtained in the previous training stage of the current training stage to obtain the features output by the current training stage.

2. The method according to claim 1, characterized in that The features of some key points removed in each training include: The features of any two adjacent key points are randomly removed.

3. The method according to claim 1, characterized in that Re-assigning weights to features of nodes on the second spatiotemporal graph in at least three different dimensions to obtain a third spatiotemporal graph, including: Inputting the features of the nodes of the second spatiotemporal graph into a spatial self-attention model, and assigning a first weight to the features of all nodes belonging to the same frame; The first weight assigned to each feature is multiplied by each feature itself and then added to each feature itself to obtain a first feature vector; Inputting the first feature vector into a temporal self-attention model, and assigning a second weight to features of the same node belonging to different frames; The second weight assigned to each feature is multiplied by each feature itself and then added to each feature itself to obtain a second feature vector; Inputting the second feature vector into the channel self-attention model, and assigning a third weight to features of the same node in different channels; The third weight assigned to each feature is multiplied by each feature itself and then added to each feature itself to obtain a third feature vector.

4. The method according to claim 1, characterized in that: The target key point data includes: skeleton features, action features, joint action features and skeleton action features, and the type of sign language contained in the image to be detected is identified based on the spatiotemporal graph structure according to the pre-trained spatiotemporal graph convolutional neural network model, including: Inputting the skeleton features, the motion features, the joint motion features and the skeleton motion features into the spatiotemporal graph convolutional neural network model, and obtaining joint feature prediction results, skeleton feature prediction results, joint motion feature prediction results and skeleton motion feature prediction results output by the spatiotemporal graph convolutional neural network model; The joint feature prediction results, the skeleton feature prediction results, the joint point motion feature prediction results and the skeleton motion feature prediction results are fused to obtain the type of sign language contained in the image to be detected.

5. The method according to claim 1, characterized in that The method further comprises: After the pre-trained spatiotemporal graph convolutional neural network model identifies the type of sign language contained in the image to be detected based on the spatiotemporal graph structure, an animated image for displaying the type of sign language is displayed according to the identified type of sign language.

6. The method according to any one of claims 1 to 5, characterized in that The target key point data includes data of human finger joint points.

7. A sign language recognition device, characterized in that: include: A detection module is used to detect the key points of the human body in the image to be detected and obtain the key points data of the human body; An extraction module, used to extract target key point data related to sign language from the key point data of the whole body of the human body; A construction module, used to construct a spatiotemporal graph structure using the target key point data; A classification module, used to identify the type of sign language contained in the image to be detected based on the space-time graph structure according to a pre-trained space-time graph convolutional neural network model, wherein in the process of training the space-time graph convolutional neural network model, the features of some key points are removed in each training; The device also includes: a training module, which is used to train the spatiotemporal graph convolutional neural network model before identifying the type of sign language contained in the image to be detected based on the spatiotemporal graph structure according to the pre-trained spatiotemporal graph convolutional neural network model, wherein the training of the spatiotemporal graph convolutional neural network model includes multiple training stages that are repeatedly executed, and each training stage includes: performing spatial graph convolution on the features of the nodes on the first spatiotemporal graph constructed based on the training data to obtain a second spatiotemporal graph; reallocating weights on the features of the nodes on the second spatiotemporal graph in at least three different dimensions to obtain a third spatiotemporal graph; performing temporal graph convolution on the features of the nodes on the third spatiotemporal graph to obtain a fourth spatiotemporal graph; removing the features of some nodes on the fourth spatiotemporal graph, and superimposing the remaining features with the features obtained in the previous training stage of the current training stage to obtain the features output by the current training stage.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the sign language recognition method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sign language recognition method according to any one of claims 1 to 6 is implemented.

Images