Gesture recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a gesture recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a frame image containing a gesture action, and generating a gesture skeleton sequence comprising a spatial position code and a time position code based on the frame image; the spatial position code is used for representing the position of each joint in the gesture skeleton, and the time position code is used for representing the relationship of each gesture skeleton in the gesture skeleton sequence, constructing a corresponding sparse directed matrix based on the gesture skeleton sequence, constructing a corresponding space-time diagram based on the sparse directed matrix, and displaying the corresponding space-time diagram. And classifying the space-time diagram to obtain a recognition result of the gesture action. According to the embodiment of the invention, the calculation efficiency and the interpretability are improved, and the problems of low precision, low efficiency and poor anti-interference capability of a man-machine gesture action recognition algorithm are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, and in particular to a gesture recognition method, a gesture recognition device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the continuous increase of vehicle ownership and traffic accident rate, urban traffic and life and property safety are facing increasingly severe challenges. In this context, automatic driving technology, as an innovative traffic solution, has higher operation efficiency and intelligent planning capability, which can effectively improve the safety of the traffic system, avoid collisions and dangerous driving behaviors, optimize traffic flow, improve traffic conditions, and promote the intelligent and green development of urban traffic systems.

[0003] Although automatic driving technology is highly regarded, the key to achieving truly intelligent driving lies in its accurate understanding of the surrounding and internal environment and intelligent decision-making capability. Among them, human-vehicle gesture interaction is an important development direction in the field of automobile intelligence. Human-vehicle gesture interaction refers to recognizing the gestures of drivers or passengers to control various functions of the vehicle, such as answering the phone, adjusting the music volume, etc., thereby improving the convenience and safety of driving, and also helping to improve driving safety.

[0004] Although existing research work has shown good results in human-vehicle gesture interaction, many problems remain to be solved. First, the existing Yolov5 model directly used for gesture recognition in the vehicle environment has poor generalization ability and robustness under uneven lighting conditions in the vehicle. Second, the spatial and temporal trajectories of hand joints are ignored, and the computational efficiency is low. SUMMARY

[0005] In view of the above problems, the present application embodiments are proposed in order to provide a gesture recognition method, a gesture recognition device, an electronic device and a computer readable storage medium which overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, the present application embodiments disclose a gesture recognition method, comprising:

[0007] obtaining a frame image containing a gesture action, and generating a gesture skeleton sequence including spatial position encoding and temporal position encoding based on the frame image; the spatial position encoding is used to represent the position of each joint in the gesture skeleton, and the temporal position encoding is used to represent the relationship between each gesture skeleton in the gesture skeleton sequence;

[0008] constructing a corresponding sparse directed matrix based on the gesture skeleton sequence;

[0009] constructing a corresponding space-time graph based on the sparse directed matrix;

[0010] classifying the spatio-temporal graph to obtain a recognition result of the gesture action.

[0011] Optionally, the generating the gesture skeleton sequence comprising spatial position encoding and temporal position encoding based on the frame image comprises:

[0012] performing gesture skeleton recognition on the frame image to obtain a gesture skeleton sequence corresponding to each frame image;

[0013] constructing a spatial domain and a temporal domain of the gesture skeleton sequence to obtain a first candidate gesture skeleton sequence; the spatial domain is used to represent spatial interaction between joints in the gesture skeleton, and the temporal domain is used to represent a situation of change of each joint in the gesture skeleton over time;

[0014] adding joint connectivity information to each joint in the candidate gesture skeleton sequence to obtain a second candidate gesture skeleton sequence; the joint connectivity information is used to represent a number of joints connected to a current joint;

[0015] allocating spatial position encoding to each joint in the second candidate gesture skeleton sequence, and allocating temporal position encoding to each gesture skeleton in the second candidate gesture skeleton sequence to obtain a gesture skeleton sequence comprising spatial position encoding and temporal position encoding.

[0016] Optionally, the constructing a corresponding sparse directed matrix based on the gesture skeleton sequence comprises:

[0017] initializing an original spatial matrix and an original temporal matrix to obtain a target spatial matrix and a target temporal matrix;

[0018] obtaining dense spatial and temporal interaction between joints in the gesture skeleton sequence by using a multi-head attention mechanism;

[0019] generating a spatial dense interaction matrix and a temporal dense interaction matrix by using the dense spatial and temporal interaction;

[0020] generating a directed spatial matrix and a directed temporal matrix by using an asymmetric convolution kernel, the spatial dense interaction matrix and the temporal dense interaction matrix;

[0021] filtering the directed spatial matrix and the directed temporal matrix to obtain a candidate sparse directed spatial matrix and a candidate sparse directed temporal matrix;

[0022] constructing a final sparse directed spatial matrix and a final sparse directed temporal matrix by using an activation function, a preset mask, the candidate sparse directed spatial matrix and the candidate sparse directed temporal matrix.

[0023] Optionally, constructing the corresponding spatiotemporal graph based on the sparse directed matrix includes:

[0024] Interaction pattern features are obtained by using space-time graph routing and the sparse directed matrix to obtain the first spatiotemporal graph;

[0025] The model interaction features are obtained by using the time-space graph routing and the sparse directed matrix to obtain the second time-space graph;

[0026] The first and second spatiotemporal diagrams are merged to obtain the final spatiotemporal diagram.

[0027] Optionally, classifying the spatiotemporal graph to obtain the recognition result of the gesture action includes:

[0028] The spatiotemporal graph is classified using mean pooling and a multilayer perceptron, and the classification result is used as the recognition result of the gesture action.

[0029] Optional, also includes:

[0030] The spatiotemporal sparse graph convolutional network is trained using the recognition results and a preset loss function to obtain the trained spatiotemporal sparse graph convolutional network, and then the trained spatiotemporal sparse circular convolutional network is used for the next gesture recognition.

[0031] Secondly, embodiments of the present invention disclose a gesture recognition device, comprising:

[0032] The acquisition module is used to acquire frame images containing gesture actions;

[0033] A generation module is used to generate a gesture skeleton sequence including spatial position encoding and temporal position encoding based on the frame image; the spatial position encoding is used to characterize the position of each joint in the gesture skeleton, and the temporal position encoding is used to characterize the relationship between each gesture skeleton in the gesture skeleton sequence;

[0034] The first construction module is used to construct a corresponding sparse directed matrix based on the gesture skeleton sequence;

[0035] The second construction module is used to construct the corresponding spatiotemporal graph based on the sparse directed matrix;

[0036] The classification module is used to classify the spatiotemporal graph to obtain the recognition result of the gesture action.

[0037] Optional, specifically used for:

[0038] Perform gesture skeleton recognition on the frame images to obtain gesture skeleton sequences corresponding to each frame image;

[0039] The spatial and temporal domains of the gesture skeleton sequence are constructed to obtain a first candidate gesture skeleton sequence; the spatial domain is used to characterize the spatial interaction between each joint in the gesture skeleton, and the temporal domain is used to characterize the changes of each joint in the gesture skeleton over time.

[0040] Joint connectivity information is added to each joint in the candidate gesture skeleton sequence to obtain a second candidate gesture skeleton sequence; the joint connectivity information is used to characterize the number of joints connected to the current joint;

[0041] Spatial position codes are assigned to each joint in the second candidate gesture skeleton sequence, and temporal position codes are assigned to each gesture skeleton in the second candidate gesture skeleton sequence, resulting in a gesture skeleton sequence that includes both spatial and temporal position codes.

[0042] Optional, specifically used for:

[0043] The original spatial matrix and the original temporal matrix are initialized to obtain the target spatial matrix and the target temporal matrix;

[0044] A multi-head attention mechanism is employed to obtain the dense spatial and temporal interactions between the joints in the gesture skeleton sequence;

[0045] The dense spatial and temporal interactions are used to generate a spatial dense interaction matrix and a temporal dense interaction matrix;

[0046] A directed spatial matrix and a directed temporal matrix are generated using an asymmetric convolution kernel, the spatially dense interaction matrix, and the temporally dense interaction matrix.

[0047] The directed spatial matrix and the directed time matrix are filtered to obtain candidate sparse directed spatial matrices and candidate sparse directed time matrices;

[0048] The final sparse directed space matrix and sparse directed time matrix are constructed by using an activation function, a preset mask, the candidate sparse directed space matrix, and the candidate sparse directed time matrix.

[0049] Optional, specifically used for:

[0050] Interaction pattern features are obtained by using space-time graph routing and the sparse directed matrix to obtain the first spatiotemporal graph;

[0051] The model interaction features are obtained by using the time-space graph routing and the sparse directed matrix to obtain the second time-space graph;

[0052] The first and second spatiotemporal diagrams are merged to obtain the final spatiotemporal diagram.

[0053] Optionally, the classification module is specifically used for:

[0054] The spatiotemporal graph is classified using mean pooling and a multilayer perceptron, and the classification result is used as the recognition result of the gesture action.

[0055] Optional, also includes:

[0056] The training module is used to train the spatiotemporal sparse graph convolutional network using the recognition results and a preset loss function to obtain the trained spatiotemporal sparse graph convolutional network, and to use the trained spatiotemporal sparse circular convolutional network for the next gesture recognition.

[0057] Thirdly, the present invention discloses an electronic device comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the gesture recognition method described above.

[0058] Fourthly, the present invention discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the gesture recognition method described above.

[0059] The embodiments of the present invention have the following advantages:

[0060] A frame image containing a gesture is acquired, and a gesture skeleton sequence including spatial and temporal location encodings is generated based on the frame image. The spatial location encoding is used to characterize the position of each joint in the gesture skeleton, and the temporal location encoding is used to characterize the relationship between the gesture skeletons in the gesture skeleton sequence. Then, a corresponding sparse directed matrix is ​​constructed based on the gesture skeleton sequence, and a corresponding spatiotemporal graph is constructed based on the sparse directed matrix. The spatiotemporal graph is then classified to obtain the recognition result of the gesture. In this way, by introducing sparse spatiotemporal directed interaction, the limitations associated with dense and undirected methods are overcome. The sparse aspect selectively simulates the basic interactions between hand joints, improving computational efficiency and interpretability, and solving the problems of low accuracy, low efficiency, and poor anti-interference ability of human-computer gesture recognition algorithms. Attached Figure Description

[0061] Figure 1 The following is a flowchart of a gesture recognition method provided in an embodiment of the present invention. Figure 1 ;

[0062] Figure 2 The following is a flowchart of a gesture recognition method provided in an embodiment of the present invention. Figure 2 ;

[0063] Figure 3This is a structural block diagram of a gesture recognition device provided in an embodiment of the present invention. Detailed Implementation

[0064] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0065] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0066] like Figure 1 As shown, this embodiment proposes a gesture recognition method that can be applied to Spatiotemporal Sparse Graph Convolutional Network (ST-SGCN) for dynamic gesture recognition of passengers inside a vehicle. Specifically, the method may include:

[0067] Step 101: Obtain a frame image containing a gesture action, and generate a gesture skeleton sequence including spatial position encoding and temporal position encoding based on the frame image; the spatial position encoding is used to characterize the position of each joint in the gesture skeleton, and the temporal position encoding is used to characterize the relationship between each gesture skeleton in the gesture skeleton sequence.

[0068] In practical applications, image acquisition devices, such as cameras, can be installed inside the vehicle to capture frame images containing the hand gestures of passengers. The number of frame images can be one or multiple; in practice, the number can be set according to actual needs, and this embodiment does not impose any limitations. For ease of description, this embodiment will be described using the assumption that there are multiple frame images.

[0069] Then, a gesture skeleton sequence is generated based on all the frame images. The gesture skeleton sequence includes spatial position coding and temporal position coding. Spatial position coding is used to represent the position of each joint in any gesture skeleton, and temporal position coding is used to represent the sequential relationship between each gesture skeleton in the gesture skeleton sequence.

[0070] In this embodiment, generating a gesture skeleton sequence including spatial location encoding and temporal location encoding based on the frame image includes:

[0071] Perform gesture skeleton recognition on the frame images to obtain gesture skeleton sequences corresponding to each frame image;

[0072] The spatial and temporal domains of the gesture skeleton sequence are constructed to obtain a first candidate gesture skeleton sequence; the spatial domain is used to characterize the spatial interaction between each joint in the gesture skeleton, and the temporal domain is used to characterize the changes of each joint in the gesture skeleton over time.

[0073] Joint connectivity information is added to each joint in the candidate gesture skeleton sequence to obtain a second candidate gesture skeleton sequence; the joint connectivity information is used to characterize the number of joints connected to the current joint;

[0074] Spatial position codes are assigned to each joint in the second candidate gesture skeleton sequence, and temporal position codes are assigned to each gesture skeleton in the second candidate gesture skeleton sequence, resulting in a gesture skeleton sequence that includes both spatial and temporal position codes.

[0075] Specifically, after acquiring frame images containing gesture actions, gesture skeleton recognition can be performed on the frame images. Since there are multiple frame images, there are multiple gesture skeletons corresponding to the recognized images. Arranging these gesture skeletons according to the order of the frame images yields a gesture skeleton sequence, which can be represented as follows: Where T is the number of frames, N is the number of joints in the gesture skeleton, and 3 represents the three-dimensional coordinates (x, y, z) of each joint. In this way, by capturing the asymmetric dependencies between joints through three-dimensional coordinates, the ability to discriminate the influence of joints can be improved.

[0076] It should be noted that gesture skeleton recognition can be achieved using MediaPipe or other methods. In practical applications, the specific method of gesture skeleton recognition can be set according to actual needs, and this embodiment does not impose any restrictions on it.

[0077] For the recognized gesture skeleton sequence, a spatial domain and a temporal domain can be constructed to obtain a gesture skeleton sequence containing both spatial and temporal domains (for ease of description, it is denoted as the "first candidate gesture skeleton sequence"). The spatial domain represents the spatial interaction between the joints in the gesture skeleton, and the temporal domain represents the changes of the joints in the gesture skeleton over time.

[0078] When constructing a spatial domain, the time dimension T and the spatial dimension N can be interchanged to obtain the spatial domain, which can be represented as follows: In this way, the positional information of each joint in different frame images is preserved, while the temporal dimension is used to capture the spatial relationships between joints. Similarly, by permuting the axes of the gesture skeleton sequence, the temporal domain is constructed, which is represented as... The temporal domain emphasizes the changes of each joint over time, that is, the interaction between frames.

[0079] For the first candidate gesture skeleton sequence, joint connectivity information, also known as centrality encoding, can be added to each joint to obtain a gesture skeleton sequence containing joint connectivity information (denoted as the "second candidate gesture skeleton sequence"). The joint connectivity information represents the number of joints connected to any current joint, which helps to enhance each joint's perception of its domain information.

[0080] For the second candidate gesture skeleton sequence, a spatial position code can be assigned to each joint. The spatial position code represents the position of the joint in the gesture skeleton, which helps the neural network understand the relative positional relationships between joints. The representation of the spatial position code can be an integer or other open forms. In practical applications, it can be set according to actual needs, and this embodiment does not impose any restrictions on it.

[0081] Since frame images themselves do not possess attributes indicating their position, a temporal position code can be assigned to each gesture skeleton in the second candidate gesture skeleton sequence to ensure that the same joint in different frame images has the same code, thereby obtaining a gesture skeleton sequence that includes both spatial and temporal position codes. The temporal position code can be performed using different frequencies of sine and cosine functions, or other methods can be used. In practical applications, the settings can be configured according to actual needs; this embodiment does not impose any limitations on this.

[0082] Step 102: Construct the corresponding sparse directed matrix based on the gesture skeleton sequence.

[0083] After obtaining the gesture skeleton sequence including spatial and temporal location encoding, a corresponding sparse directed matrix can be constructed, thereby overcoming the limitations associated with dense and undirected methods.

[0084] In this embodiment, constructing the corresponding sparse directed matrix based on the gesture skeleton sequence includes:

[0085] The original spatial matrix and the original temporal matrix are initialized to obtain the target spatial matrix and the target temporal matrix;

[0086] A multi-head attention mechanism is employed to obtain the dense spatial and temporal interactions between the joints in the gesture skeleton sequence;

[0087] The dense spatial and temporal interactions are used to generate a spatial dense interaction matrix and a temporal dense interaction matrix;

[0088] A directed spatial matrix and a directed temporal matrix are generated using an asymmetric convolution kernel, the spatially dense interaction matrix, and the temporally dense interaction matrix.

[0089] The directed spatial matrix and the directed time matrix are filtered to obtain candidate sparse directed spatial matrices and candidate sparse directed time matrices;

[0090] The final sparse directed space matrix and sparse directed time matrix are constructed by using an activation function, a preset mask, the candidate sparse directed space matrix, and the candidate sparse directed time matrix.

[0091] Specifically, for the preset original spatial matrix, it can be initialized based on the natural topological structure of the hand. For example, based on knowledge of hand anatomy, it can predefine which joints are interconnected, thereby obtaining the target spatial matrix A. s Furthermore, the preset original time matrix can be initialized as an upper triangular matrix, representing the influence of subsequent frame images on the previous frame image in the time series, thereby obtaining the target time matrix A. t This initialization method assumes that, in a time series, subsequent gestures may be influenced by previous gestures.

[0092] A multi-head attention mechanism is used to capture dense spatial and temporal interactions between joints in a gesture skeleton sequence. Since dense spatial and temporal interactions involve calculating the influence weights of each joint on other joints, a spatial dense interaction matrix and a temporal dense interaction matrix can be generated based on the dense spatial and temporal interactions. The spatial dense interaction matrix is ​​represented as follows: The time-dense interaction matrix is ​​represented as Attention mechanisms allow neural networks to learn which interactions between joints are more important, thus providing a foundation for subsequent sparsification and directionation.

[0093] By applying asymmetric convolution kernels to the temporal and spatial domains, and combining spatially and temporally dense interaction matrices, a directed spatial matrix and a directed temporal matrix are generated. The directed spatial matrix is ​​denoted as... The directed time matrix is ​​denoted as The choice of asymmetric convolution kernels is based on the assumption that the effect of one joint on another may differ from its reverse effect. This allows for better capture of asymmetry in hand gestures.

[0094] Through a preset binary mask M s and M tThe directed spatial matrix and the directed temporal matrix are filtered to remove matrices with scores below a threshold ∈ , thus obtaining the filtered sparse directed spatial matrix (denoted as the "candidate sparse directed spatial matrix") and sparse directed temporal matrix (denoted as the "candidate sparse directed temporal matrix"). The candidate sparse directed spatial matrix is ​​denoted as . The candidate sparse directed time matrix is ​​denoted as This reduces unnecessary interactions, thereby lowering computational complexity and improving the interpretability of neural networks.

[0095] The binary mask can be generated using the following formula:

[0096]

[0097] It should be noted that, in addition to binary masks, other masks can also be used for the preset mask. In practical applications, the specific form of the preset mask can be set according to actual needs, and this embodiment does not impose any restrictions on this.

[0098] In obtaining and Then, activation functions can be used to further enhance the sparsity of the matrix. The activation function can be expressed by the following formula:

[0099]

[0100] Here, x is the input value, and ∈ is a constant used for stability. The characteristic of this activation function is that it maps the input values ​​to a sparse output, where most values ​​are close to zero, and only a few are significantly greater than zero.

[0101] It should be noted that, in addition to the above formula, other activation functions can also be used. In practical applications, the specific activation function can be set according to actual needs, and this embodiment does not impose any restrictions on this.

[0102] Then, the final sparse directed space matrix is ​​constructed using the activation function, a preset mask, candidate sparse directed space matrices, and candidate sparse directed time matrices. and sparse directed time matrix The sparse directed spatial matrix and the sparse directed temporal matrix represent the spatial and temporal relationships between joints, respectively. The specific construction method is as follows:

[0103]

[0104] Here, ⊙ represents element-wise multiplication.

[0105] Step 103: Construct the corresponding spatiotemporal graph based on the sparse directed matrix.

[0106] After constructing the sparse directed matrix, the corresponding spatiotemporal graph can be further constructed based on the sparse directed matrix to facilitate subsequent classification of the spatiotemporal graph and obtain the classification result.

[0107] In this embodiment, constructing the corresponding spatiotemporal graph based on the sparse directed matrix includes:

[0108] Interaction pattern features are obtained by using space-time graph routing and the sparse directed matrix to obtain the first spatiotemporal graph;

[0109] The model interaction features are obtained by using the time-space graph routing and the sparse directed matrix to obtain the second time-space graph;

[0110] The first and second spatiotemporal diagrams are merged to obtain the final spatiotemporal diagram.

[0111] Specifically, Space-Time Graph Routing (STGR) can be used to capture interaction pattern features, that is, the spatial and temporal interactions between joints. Therefore, STGR can be used to capture interaction pattern features of sparse directed spatial matrices and sparse directed temporal matrices, resulting in a spatiotemporal graph (denoted as the "first spatiotemporal graph"). This can be calculated using the following formula:

[0112]

[0113] in, This is the graph representation of the (l-1)th layer, W s1 and W t1 δ is the weight matrix of the graph convolutional network, and δ is a non-linear activation function, such as ReLU. Of course, it can also be other non-linear activation functions. In practical applications, it can be set according to actual needs. This embodiment does not limit it.

[0114] Furthermore, Temporal-Space Graph Routing (TSGR) can be used to capture pattern interaction features, that is, the temporal variations and spatial relationships of joints. Therefore, TSGR can be used to capture pattern interaction features of sparse directed spatial matrices and sparse directed temporal matrices, resulting in a spatiotemporal graph (denoted as the "second spatiotemporal graph"). Specifically, it can be calculated using the following formula:

[0115]

[0116] in, This is the graph representation of the (l-1)th layer, W t2 and W s2 It is the weight matrix of the graph convolutional network.

[0117] Then, the first and second spatiotemporal graphs are merged to reconstruct the global spatiotemporal graph. That is, the representations of the two spatiotemporal graphs are reshaped to the same dimensions, and the two are added together or fused in other ways to form a comprehensive spatiotemporal graph representation that contains both spatial and temporal information.

[0118] Step 104: Classify the spatiotemporal graph to obtain the recognition result of the gesture action.

[0119] After obtaining the merged spatiotemporal graph, it can be classified to obtain the classification result, which is then used as the recognition result for gesture actions.

[0120] In this embodiment, classifying the spatiotemporal graph to obtain the recognition result of the gesture includes:

[0121] The spatiotemporal graph is classified using mean pooling and a multilayer perceptron, and the classification result is used as the recognition result of the gesture action.

[0122] Specifically, for the merged spatiotemporal graph, mean pooling and multilayer perceptron (MLP) can be used to classify it, and the classification result can be used as the recognition result of gesture action.

[0123] It should be noted that before classifying the merged spatiotemporal graph, some additional processing can be performed, such as pooling, to reduce the dimensionality of features and extract relevant information. Of course, in practical applications, whether additional processing is needed, and the specific processing method, can be set according to actual needs; this embodiment does not impose any restrictions on this.

[0124] In this embodiment, frame images containing gesture actions are acquired, and a gesture skeleton sequence including spatial and temporal location encodings is generated based on the frame images. The spatial location encoding is used to characterize the position of each joint in the gesture skeleton, and the temporal location encoding is used to characterize the relationship between each gesture skeleton in the gesture skeleton sequence. Then, a corresponding sparse directed matrix is ​​constructed based on the gesture skeleton sequence, and a corresponding spatiotemporal graph is constructed based on the sparse directed matrix. The spatiotemporal graph is then classified to obtain the recognition result of the gesture action. In this way, by introducing sparse spatiotemporal directed interaction, the limitations associated with dense and undirected methods are overcome. The sparse aspect selectively simulates the basic interactions between hand joints, improving computational efficiency and interpretability, and solving the problems of low accuracy, low efficiency, and poor anti-interference ability of human-computer gesture action recognition algorithms.

[0125] like Figure 2As shown, this embodiment proposes another gesture recognition method, which can be applied to Spatiotemporal Sparse Graph Convolutional Networks (ST-SGCN) for dynamic gesture recognition. Specifically, this method may include:

[0126] Step 201: Obtain a frame image containing a gesture action, and generate a gesture skeleton sequence including spatial position encoding and temporal position encoding based on the frame image; the spatial position encoding is used to characterize the position of each joint in the gesture skeleton, and the temporal position encoding is used to characterize the relationship between each gesture skeleton in the gesture skeleton sequence.

[0127] Step 202: Construct the corresponding sparse directed matrix based on the gesture skeleton sequence;

[0128] Step 203: Construct the corresponding spatiotemporal graph based on the sparse directed matrix;

[0129] Step 204: Classify the spatiotemporal graph to obtain the recognition result of the gesture action.

[0130] Steps 201 to 204 are essentially the same as steps 101 to 104, and will not be repeated here to avoid repetition.

[0131] Step 205: The spatiotemporal sparse graph convolutional network is trained using the recognition results and a preset loss function to obtain the trained spatiotemporal sparse graph convolutional network, and the trained spatiotemporal sparse circular convolutional network is used for the next gesture recognition.

[0132] Specifically, after obtaining the recognition result, the recognition result can be compared with the actual result, and the preset loss function of the two can be calculated. Then, the spatiotemporal sparse graph convolutional network is trained through the preset loss function to obtain the trained spatiotemporal sparse graph convolutional network. In this way, the trained spatiotemporal sparse graph convolutional network can be used for the next gesture recognition.

[0133] Training can be terminated when the preset loss function meets the preset conditions, and the final spatiotemporal sparse graph convolutional network can be obtained.

[0134] It should be noted that the preset loss function can be the cross-entropy loss function or other loss functions. In practical applications, the specific loss function can be set according to actual needs, and this embodiment does not impose any restrictions on it.

[0135] Reference Figure 3 The diagram shows a structural block diagram of a gesture recognition device according to an embodiment of the present invention. The device includes:

[0136] The acquisition module 301 is used to acquire frame images containing gesture actions;

[0137] The generation module 302 is used to generate a gesture skeleton sequence including spatial position encoding and temporal position encoding based on the frame image; the spatial position encoding is used to characterize the position of each joint in the gesture skeleton, and the temporal position encoding is used to characterize the relationship between each gesture skeleton in the gesture skeleton sequence;

[0138] The first construction module 303 is used to construct a corresponding sparse directed matrix based on the gesture skeleton sequence;

[0139] The second construction module 304 is used to construct the corresponding spatiotemporal graph based on the sparse directed matrix;

[0140] The classification module 305 is used to classify the spatiotemporal graph to obtain the recognition result of the gesture action.

[0141] In this embodiment, the generation module is specifically used for:

[0142] Perform gesture skeleton recognition on the frame images to obtain gesture skeleton sequences corresponding to each frame image;

[0143] The spatial and temporal domains of the gesture skeleton sequence are constructed to obtain a first candidate gesture skeleton sequence; the spatial domain is used to characterize the spatial interaction between each joint in the gesture skeleton, and the temporal domain is used to characterize the changes of each joint in the gesture skeleton over time.

[0144] Joint connectivity information is added to each joint in the candidate gesture skeleton sequence to obtain a second candidate gesture skeleton sequence; the joint connectivity information is used to characterize the number of joints connected to the current joint;

[0145] Spatial position codes are assigned to each joint in the second candidate gesture skeleton sequence, and temporal position codes are assigned to each gesture skeleton in the second candidate gesture skeleton sequence, resulting in a gesture skeleton sequence that includes both spatial and temporal position codes.

[0146] In this embodiment, it is specifically used for:

[0147] The original spatial matrix and the original temporal matrix are initialized to obtain the target spatial matrix and the target temporal matrix;

[0148] A multi-head attention mechanism is employed to obtain the dense spatial and temporal interactions between the joints in the gesture skeleton sequence;

[0149] The dense spatial and temporal interactions are used to generate a spatial dense interaction matrix and a temporal dense interaction matrix;

[0150] A directed spatial matrix and a directed temporal matrix are generated using an asymmetric convolution kernel, the spatially dense interaction matrix, and the temporally dense interaction matrix.

[0151] The directed spatial matrix and the directed time matrix are filtered to obtain candidate sparse directed spatial matrices and candidate sparse directed time matrices;

[0152] The final sparse directed space matrix and sparse directed time matrix are constructed by using an activation function, a preset mask, the candidate sparse directed space matrix, and the candidate sparse directed time matrix.

[0153] In this embodiment, the second building module is specifically used for:

[0154] Interaction pattern features are obtained by using space-time graph routing and the sparse directed matrix to obtain the first spatiotemporal graph;

[0155] The model interaction features are obtained by using the time-space graph routing and the sparse directed matrix to obtain the second time-space graph;

[0156] The first and second spatiotemporal diagrams are merged to obtain the final spatiotemporal diagram.

[0157] In this embodiment, the classification module is specifically used for:

[0158] The spatiotemporal graph is classified using mean pooling and a multilayer perceptron, and the classification result is used as the recognition result of the gesture action.

[0159] In this embodiment, it also includes:

[0160] The training module is used to train the spatiotemporal sparse graph convolutional network using the recognition results and a preset loss function to obtain the trained spatiotemporal sparse graph convolutional network, and to use the trained spatiotemporal sparse circular convolutional network for the next gesture recognition.

[0161] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0162] This invention also provides an electronic device, comprising:

[0163] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described gesture recognition method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0164] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described gesture recognition method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0165] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0171] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0172] The foregoing has provided a detailed description of a gesture recognition method, a gesture recognition device, an electronic device, and a computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A gesture recognition method, characterized by, The method comprises: acquiring frame images containing gesture actions, and generating gesture skeleton sequences comprising spatial position encodings and temporal position encodings based on the frame images; the spatial position encodings are used to represent the positions of joints in the gesture skeletons, and the temporal position encodings are used to represent the relationships between gesture skeletons in the gesture skeleton sequences; constructing a corresponding sparse directed matrix based on the gesture skeleton sequences; constructing a corresponding space-time graph based on the sparse directed matrix; classifying the space-time graph to obtain a recognition result of the gesture action.

2. The gesture recognition method of claim 1, wherein, The gesture skeleton sequences comprising spatial position encodings and temporal position encodings generated based on the frame images comprise: performing gesture skeleton recognition on the frame images to obtain gesture skeleton sequences corresponding to each frame image; constructing a spatial domain and a temporal domain of the gesture skeleton sequences to obtain first candidate gesture skeleton sequences; the spatial domain is used to represent the spatial interaction between joints in the gesture skeletons, and the temporal domain is used to represent the change of joints in the gesture skeletons over time; adding joint connectivity information to each joint in the candidate gesture skeleton sequences to obtain second candidate gesture skeleton sequences; the joint connectivity information is used to represent the number of joints connected to the current joint; assigning spatial position encodings to each joint in the second candidate gesture skeleton sequences, and assigning temporal position encodings to each gesture skeleton in the second candidate gesture skeleton sequences to obtain gesture skeleton sequences comprising spatial position encodings and temporal position encodings.

3. The gesture recognition method of claim 1, wherein, The sparse directed matrix corresponding to the gesture skeleton sequences is constructed by: initializing an original spatial matrix and an original temporal matrix to obtain a target spatial matrix and a target temporal matrix; obtaining dense space and time interactions between joints in the gesture skeleton sequences using a multi-head attention mechanism; generating a spatial dense interaction matrix and a temporal dense interaction matrix using the dense space and the time interactions; generating a directed spatial matrix and a directed temporal matrix using an asymmetric convolution kernel, the spatial dense interaction matrix and the temporal dense interaction matrix; filtering the directed spatial matrix and the directed temporal matrix to obtain a candidate sparse directed spatial matrix and a candidate sparse directed temporal matrix; constructing a final sparse directed spatial matrix and a final sparse directed temporal matrix using an activation function, a preset mask, the candidate sparse directed spatial matrix and the candidate sparse directed temporal matrix.

4. The gesture recognition method of claim 1, wherein, The space-time graph corresponding to the sparse directed matrix is constructed by: obtaining interaction mode features using a space-time graph routing and the sparse directed matrix to obtain a first space-time graph; obtaining model interaction features using a time-space graph routing and the sparse directed matrix to obtain a second space-time graph; merging the first space-time graph and the second space-time graph to obtain a final space-time graph.

5. The gesture recognition method of claim 1, wherein, The space-time graph is classified to obtain a recognition result of the gesture action, which comprises: classifying the space-time graph using mean pooling and a multilayer perceptron, and taking the classification result as the recognition result of the gesture action.

6. The gesture recognition method of claim 1, wherein, It also comprises: The spatio-temporal sparse graph convolutional network is trained by using the recognition result and a preset loss function, to obtain a trained spatio-temporal sparse graph convolutional network, and the trained spatio-temporal sparse graph convolutional network is used for gesture action recognition in the next time.

7. A gesture recognition apparatus, characterized by The method comprises the steps of: acquiring a frame image containing a gesture action; generating a gesture skeleton sequence comprising spatial position encoding and temporal position encoding based on the frame image; the spatial position encoding is used to represent the positions of joints in the gesture skeleton, and the temporal position encoding is used to represent the relationship between gesture skeletons in the gesture skeleton sequence; constructing a corresponding sparse directed matrix based on the gesture skeleton sequence by a first constructing module; constructing a corresponding spatio-temporal graph based on the sparse directed matrix by a second constructing module; classifying the spatio-temporal graph to obtain a recognition result of the gesture action.

8. The gesture recognition apparatus of claim 7, wherein, The generating module is specifically configured to: perform gesture skeleton recognition on the frame image to obtain a gesture skeleton sequence corresponding to each frame image; construct a spatial domain and a temporal domain of the gesture skeleton sequence to obtain a first candidate gesture skeleton sequence; the spatial domain is used to represent the spatial interaction between joints in the gesture skeleton, and the temporal domain is used to represent the change of the joints in the gesture skeleton over time; add joint connectivity information to each joint in the candidate gesture skeleton sequence to obtain a second candidate gesture skeleton sequence; the joint connectivity information is used to represent the number of joints connected to the current joint; assign spatial position encoding to each joint in the second candidate gesture skeleton sequence, and assign temporal position encoding to each gesture skeleton in the second candidate gesture skeleton sequence to obtain a gesture skeleton sequence comprising spatial position encoding and temporal position encoding.

9. An electronic device, comprising: The method comprises the steps of: a processor, a memory, and a computer program stored on the memory and capable of running on the processor, wherein the computer program is executed by the processor to implement the steps of the gesture recognition method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the gesture recognition method according to any one of claims 1-6.

Images