Gesture recognition method and device, storage medium and electronic device

Abstract

The present invention provides a gesture recognition method and device, a storage medium and an electronic device, the method comprising: obtaining a preprocessed skeleton sequence; determining the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence; determining the joint point set distance feature and the joint point set direction feature of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image; calling a preset feature extraction module to extract the motion trajectory feature of the skeleton sequence; processing the motion trajectory feature of the skeleton sequence, the joint point distance feature and the joint point set direction feature of each frame of the skeleton image, and obtaining the gesture recognition result of the skeleton sequence. By using the coordinates of the joint points, the joint point set distance feature and the joint point set method feature of the skeleton image can be quickly extracted. The whole process does not need to build a complex and computationally intensive model, which reduces the complexity and computational complexity of gesture recognition, so that it can be applied to low-performance, low-power intelligent devices and reduce the cost of gesture recognition.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a gesture recognition method and device, a storage medium and an electronic device. Background Art

[0002] Human-computer interaction is becoming more and more common in life. Compared with other data, skeleton gesture data is more compact and environmental condition data is more robust. Therefore, gesture recognition has become one of the commonly used methods in human-computer interaction applications.

[0003] At present, there are two types of research on gesture recognition based on hand skeletons: one is the traditional method based on hand-designed features, and the other is the method based on deep learning. When using the traditional deep learning method to recognize hand skeleton gestures, it is necessary to build an extremely complex and computationally intensive recognition model, and to extract extremely complex features. Traditional gesture recognition methods are only supported in high-performance, high-power computing devices, which increases the application cost of gesture recognition. Summary of the invention

[0004] In view of this, the present invention provides a gesture recognition method and device, a storage medium and an electronic device. The gesture recognition method provided by the present invention reduces the application of complex neural networks. The distance features and direction features of the joint point sets of the skeleton diagram can be determined through the coordinates of the joint points, which reduces the amount of calculation in the gesture recognition process. It can be applied in low-power computing devices to reduce application costs.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of the present invention discloses a gesture recognition method, which is applied to a gesture recognition device, comprising:

[0007] Get the preprocessed skeleton sequence;

[0008] Determine the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence;

[0009] Based on the coordinates of each joint point of the skeleton image of each frame, determining the joint point set distance feature of the skeleton image of each frame;

[0010] Based on the coordinates of each joint point of the skeleton image of each frame, determining the directional features of the joint point set of the skeleton image of each frame;

[0011] Calling a preset feature extraction module to extract motion trajectory features of the skeleton sequence;

[0012] The motion trajectory features of the skeleton sequence, the joint point set distance features of the skeleton graph of each frame, and the joint point direction features are processed to obtain a gesture recognition result of the skeleton sequence.

[0013] In the above method, optionally, the skeleton sequence preprocessing process includes:

[0014] Collect the video to be identified;

[0015] Extracting an initial skeleton sequence from the video to be identified;

[0016] Performing denoising on the initial skeleton sequence to obtain a denoised skeleton sequence;

[0017] The denoised skeleton sequence is normalized to complete the preprocessing of the skeleton sequence.

[0018] In the above method, optionally, determining the joint point set distance feature of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image comprises:

[0019] For each frame of the skeleton graph, each joint point in the skeleton graph of the frame is combined in pairs to obtain a plurality of first joint point groups;

[0020] For each frame of the skeleton image, the coordinates of each joint point in the skeleton image of the frame are applied to determine the Euclidean distance of each first joint point group of the skeleton image of the frame, and each Euclidean distance is processed to obtain the joint point set distance feature of the skeleton image of the frame.

[0021] In the above method, optionally, determining the directional features of the joint point set of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image comprises:

[0022] Selecting a starting joint point from each joint point of the skeleton graph in each frame;

[0023] For each frame of the skeleton image, matching the starting joint point of the skeleton image with each remaining joint point in the skeleton image to obtain a plurality of second joint point groups;

[0024] Determine the offset of each second joint point group of the skeleton image of each frame based on the coordinates of each joint point of the skeleton image of each frame;

[0025] Based on each offset of the skeleton image of each frame, a direction cone feature of a joint point of the skeleton image of each frame is obtained;

[0026] Determine an undirected graph of the skeleton graph of each frame, wherein the undirected graph is composed of various joint points of the skeleton graph;

[0027] For each frame of the skeleton graph, the coordinates of each joint point of the skeleton graph are applied to process the undirected graph of the skeleton graph to obtain a directional tree feature of the joint points of the skeleton graph;

[0028] Based on the joint point direction cone features and the joint point direction tree features of the skeleton image of each frame, the joint point set direction features of the skeleton image of each frame are obtained.

[0029] In the above method, optionally, the coordinates of each joint point of the skeleton graph are applied to process the undirected graph of the skeleton graph to obtain the direction tree features of the joint points of the skeleton graph, including:

[0030] Creating a first node list, a second node list, and a feature list;

[0031] Select a root node from each node of the undirected graph, and add the root node to the first node list and the second node list;

[0032] Determine whether the first node list is empty;

[0033] If it is determined that the first node list is not empty, outputting the target joint point from the first node list;

[0034] Based on the second node list and the target joint point, determining whether there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph;

[0035] If it is determined that there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph, each undirected edge corresponding to the target joint point is determined as a target undirected edge;

[0036] Based on the coordinates of the joint points of each of the target undirected edges, determine the vector of each of the target undirected edges, and add each of the vectors to the feature list;

[0037] Based on the two joint points of each of the target undirected edges, update the first node list and the second node list, and return to execute the step of determining whether the first node list is empty;

[0038] If it is determined that the first node list is empty, then based on each vector in the feature list, a joint point direction tree feature of the skeleton graph is obtained.

[0039] The above method may optionally further include:

[0040] If it is determined that there is no undirected edge corresponding to the target joint point in the edge set of the undirected graph, return to the step of determining whether the node list is empty.

[0041] A second aspect of the present invention discloses a gesture recognition device, comprising:

[0042] An acquisition unit, used for acquiring a preprocessed skeleton sequence;

[0043] A first determining unit, used to determine the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence;

[0044] A second determining unit is used to determine the joint point set distance feature of each frame of the skeleton image based on the coordinates of each joint point of the skeleton image of each frame;

[0045] A third determining unit is used to determine the direction features of the joint point set of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image;

[0046] An extraction unit, used to call a preset feature extraction module to extract motion trajectory features of the skeleton sequence;

[0047] The output unit is used to process the motion trajectory features of the skeleton sequence, the joint point set distance features and the joint point direction features of the skeleton image of each frame to obtain the gesture recognition result of the skeleton sequence.

[0048] The above device may optionally further include:

[0049] A collection unit, used for collecting videos to be identified;

[0050] A first extraction unit, used to extract an initial skeleton sequence from the video to be identified;

[0051] A denoising unit, used for performing denoising processing on the initial skeleton sequence to obtain a denoised skeleton sequence;

[0052] The normalization unit is used to normalize the denoised skeleton sequence to complete the preprocessing of the skeleton sequence.

[0053] In the above device, optionally, the second determining unit includes:

[0054] A combining module, for each frame of the skeleton graph, combining each joint point in the skeleton graph in pairs to obtain a plurality of first joint point groups;

[0055] The first determination module is used to apply the coordinates of each joint point in the skeleton image of each frame to determine the Euclidean distance of each first joint point group of the skeleton image of the frame, and process each Euclidean distance to obtain the joint point set distance feature of the skeleton image of the frame.

[0056] In the above device, optionally, the third determining unit includes:

[0057] A selection module, used for selecting a starting joint point from each joint point of the skeleton graph in each frame;

[0058] A matching module, for matching the starting joint point of the skeleton image with each remaining joint point in the skeleton image for each frame, to obtain a plurality of second joint point groups;

[0059] A second determination module is used to determine the offset of each second joint point group of the skeleton image of each frame based on the coordinates of each joint point of the skeleton image of each frame;

[0060] A first obtaining module is used to obtain the direction cone features of the joint points of the skeleton image of each frame based on the offsets of the skeleton image of each frame;

[0061] A third determination module is used to determine an undirected graph of the skeleton graph of each frame, wherein the undirected graph is composed of various joint points of the skeleton graph;

[0062] A second acquisition module is used for processing the undirected graph of the skeleton graph by applying the coordinates of each joint point of the skeleton graph for each frame of the skeleton graph to obtain the joint point direction tree features of the skeleton graph;

[0063] The third obtaining module is used to obtain the joint point set direction features of each frame of the skeleton image based on the joint point direction cone features and the joint point direction tree features of each frame of the skeleton image.

[0064] In the above device, optionally, the second obtaining module includes:

[0065] A creation subunit is used to create a first node list, a second node list and a feature list;

[0066] A selection subunit, configured to select a root node from among the nodes of the undirected graph, and add the root node to the first node list and the second node list;

[0067] A first judging subunit, used to judge whether the first node list is empty;

[0068] an output subunit, configured to output a target joint point from the first node list if it is determined that the first node list is not empty;

[0069] A second judgment subunit is used to judge whether there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph based on the second node list and the target joint point;

[0070] A first determining subunit is configured to determine each undirected edge corresponding to the target joint point as a target undirected edge if it is determined that there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph;

[0071] A second determining subunit is used to determine the vector of each target undirected edge based on the coordinates of the joint point of each target undirected edge, and add each vector to the feature list;

[0072] An updating subunit, used for updating the first node list and the second node list based on the two joint points of each of the target undirected edges, and triggering the first judging subunit to execute a step of judging whether the first node list is empty;

[0073] The obtaining subunit is used to obtain the joint point direction tree features of the skeleton graph based on each vector in the feature list if it is determined that the first node list is empty.

[0074] The above-mentioned device, optionally, triggers the first judgment subunit to execute the step of judging whether the node list is empty when the second judgment subunit determines that there is no undirected edge corresponding to the target joint point in the edge set of the undirected graph.

[0075] A third aspect of the present invention discloses a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the gesture recognition method as described above.

[0076] A fourth aspect of the present invention discloses an electronic device, comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to perform the gesture recognition method as described above.

[0077] Compared with the prior art, the present invention has the following advantages:

[0078] The present invention provides a gesture recognition method and device, a storage medium and an electronic device, which is applied to a gesture recognition device, and includes: obtaining a preprocessed skeleton sequence; determining the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence; determining the joint point set distance feature and the joint point set direction feature of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image; calling a preset feature extraction module to extract the motion trajectory feature of the skeleton sequence; processing the motion trajectory feature of the skeleton sequence, the joint point distance feature and the joint point set direction feature of each frame of the skeleton image, and obtaining the gesture recognition result of the skeleton sequence. By using the coordinates of the joint points, the joint point set distance feature and the joint point set method feature of the skeleton image can be quickly extracted, and the whole process does not need to build a complex and computationally intensive model, which reduces the complexity and computational complexity of gesture recognition, so that it can be applied to low-performance, low-power intelligent devices and reduce the cost of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0080] Figure 1 A method flow chart of a gesture recognition method provided by an embodiment of the present invention;

[0081] Figure 2 A flow chart of a method for determining distance features of joint point sets of a skeleton graph provided by an embodiment of the present invention;

[0082] Figure 3 A flow chart of a method for determining directional features of a joint point set of each frame of a skeleton graph provided by an embodiment of the present invention;

[0083] Figure 4 A scene example diagram of the direction cone features of the joint points of the skeleton graph provided by the embodiment of the present invention;

[0084] Figure 5 A scene example diagram of the joint point direction tree feature of the skeleton graph provided by an embodiment of the present invention;

[0085] Figure 6 A flow chart of a method for determining a joint point direction tree feature of a skeleton graph provided by an embodiment of the present invention;

[0086] Figure 7 A flowchart of the execution of the gesture recognition model provided by the embodiment of the present invention;

[0087] Figure 8 A schematic diagram of the structure of a gesture recognition device provided by an embodiment of the present invention;

[0088] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0091] With the popularity of low-cost motion sensors and the rapid development of efficient gesture estimation algorithms, as well as the advantages of skeleton data in having more compact information and being more robust to environmental conditions compared to other types of data, research on gesture recognition based on hand skeletons has attracted the attention of more and more researchers. We can roughly divide the research on gesture recognition based on hand skeletons into two types: one is the traditional method based on hand-designed features, and the other is the method based on deep learning.

[0092] With the development of machine learning, gesture recognition methods based on deep learning have become the main direction of researchers. Such methods often use an end-to-end approach to achieve gesture recognition. Since graph convolution has made good progress in node classification and graph classification, many papers use graph convolution to extract skeleton features for gesture recognition. Since graph convolution can only extract features in the spatial domain, but for dynamic gesture recognition, temporal information is also very important. To solve this problem, a model called spatiotemporal graph convolution is used to extract the spatial and temporal features of the skeleton at the same time. Or use graph convolution and self-attention mechanism to extract the features of joints and bones in space and time. However, in addition to the fact that graph convolution requires more computation than ordinary convolution, some joints are far apart on the skeleton graph, but have strong correlation in actual motion, which often limits the skeleton features learned by graph convolution.

[0093] Gesture recognition is a basic research in the human-computer interaction scenario, and is therefore widely studied in the field of computer vision. In recent years, with the popularization of cheap motion sensors and the development of efficient gesture estimation algorithms, skeleton-based gesture recognition has attracted more and more attention from researchers. Compared with other video data, skeleton data can provide more compact and environmentally robust information. Therefore, using skeletons to achieve more accurate gesture recognition has great potential application value in human-computer interaction, human behavior understanding, and monitoring systems.

[0094] Currently, gesture recognition methods are highly complex and can only be deployed on high-power computing devices. They are difficult to use in various application scenarios using low-power computing devices. Traditional gesture recognition methods have poor applicability.

[0095] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.

[0096] The method provided in the embodiment of the present invention is applicable to a gesture recognition device, which can be mounted on a computing device. Preferably, the computing device can be a low-power, low-performance device. Furthermore, a gesture recognition model is provided in the gesture recognition device.

[0097] Reference Figure 1 , is a method flow chart of a gesture recognition method provided by an embodiment of the present invention, and is specifically described as follows:

[0098] S101: Obtain a preprocessed skeleton sequence.

[0099] In the method provided by the embodiment of the present invention, the skeleton sequence received by the gesture recognition model is preprocessed, and the skeleton sequence includes multiple frames of skeleton images, and each frame of the skeleton image includes a hand skeleton.

[0100] Preferably, the preprocessing process of the skeleton sequence is further described:

[0101] Collect the video to be identified;

[0102] Extract the initial skeleton sequence from the video to be recognized;

[0103] Performing denoising on the initial skeleton sequence to obtain a denoised skeleton sequence;

[0104] The denoised skeleton sequence is normalized to complete the preprocessing of the skeleton sequence.

[0105] In the method provided by the embodiment of the present invention, an RGB camera can be used to capture video, and the captured video is the video to be identified; the MediaPipe tool is used to extract the initial skeleton sequence from the video to be identified, and it should be noted that the initial skeleton sequence contains multiple frames of skeleton graphs. After the initial skeleton sequence is extracted from the video to be identified, a median filter is used to reduce noise in each dimension of the data in the skeleton sequence, and the kernel size of the preferred median filter is 3; then the skeleton sequence is sampled into a sequence of fixed length, such as a sequence of 32 frames, using a cubic spline interpolation algorithm for each dimension, and the median filter is used again for noise reduction. Finally, the skeleton sequence is normalized to complete the preprocessing of the skeleton sequence, thereby obtaining a preprocessed skeleton sequence. Preferably, when the skeleton sequence is normalized, the mean of each dimension of the skeleton sequence is subtracted, which does not affect the calculation of the features of each joint point in the skeleton graph of each frame in the skeleton sequence, and can also achieve the effect of moving the skeleton sequence to the midpoint position for normalization.

[0106] S102: Determine the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence.

[0107] It should be noted that the coordinates of each joint point of each frame of the skeleton graph are represented by three coordinate values ​​of the Euclidean space.

[0108] S103: Based on the coordinates of each joint point of each frame of the skeleton image, determine the joint point set distance feature of each frame of the skeleton image.

[0109] Reference Figure 2 , which is a flow chart of a method for determining the distance feature of a joint point set of a skeleton graph provided by an embodiment of the present invention, and is specifically described as follows:

[0110] S201: For each frame of the skeleton graph, each joint point in the frame of the skeleton graph is combined in pairs to obtain a plurality of first joint point groups.

[0111] In the method provided by the embodiment of the present invention, each frame of the skeleton image contains a plurality of joint points. Preferably, the number of joint points in each frame of the skeleton image is the same.

[0112] For each frame of the skeleton image, the joint points in the skeleton image are combined in pairs to obtain multiple first joint point groups; exemplarily, the skeleton image contains 5 joint points, namely A1, A2, A3, A4, and A5; the obtained first joint point groups are: (A1, A2), (A1, A3), (A1, A4), (A1, A5), (A2, A3), (A2, A4), (A2, A5), (A3, A4), (A3, A5), and (A4, A5).

[0113] S202: For each frame of the skeleton image, the coordinates of each joint point in the frame of the skeleton image are applied to determine the Euclidean distance of each first joint point group of the frame of the skeleton image, and each Euclidean distance is processed to obtain the joint point set distance feature of the frame of the skeleton image.

[0114] Preferably, the coordinates of the joint points in the skeleton graph can be expressed as: J=(x, y, z), wherein x, y, z are coordinate values ​​in the Euclidean space.

[0115] Preferably, assuming that there are T frames of skeleton images in the skeleton sequence, each frame of the skeleton image has N joint points, and the skeleton image of the kth frame is used for illustration.

[0116] The coordinates of the joint point n of the skeleton graph of the kth frame can be expressed as: Preferably, n can be the number or identification of the joint point, and the value range can be a positive integer; k represents the specific frame number, and the value range can be an integer; further, assuming that the first joint point group includes the joint point and joint points Then the Euclidean distance of the first joint point group can be expressed as: Preferably, the Euclidean distance is the Euclidean distance between two joint points.

[0117] Furthermore, after obtaining the Euclidean distance of each first joint point group, a symmetric matrix containing all distance information can be obtained. In order to compress repeated information, the elements of the lower triangle of the matrix are used to form a vector, thereby obtaining the joint point set distance features of the skeleton graph.

[0118] Exemplarily, the joint point set distance feature of the skeleton graph of the kth frame can be expressed as: Among them, JCD k represents the joint point set distance feature of the k-th frame skeleton image. Preferably, the joint point set distance feature can be flattened into a length of

[0119] S104: Based on the coordinates of each joint point of each frame of the skeleton image, determine the directional features of the joint point set of each frame of the skeleton image.

[0120] It should be noted that the key point set directional features are composed of the offsets between the joint points in the skeleton graph.

[0121] refer to Figure 3 , which is a flow chart of a method for determining the directional features of the joint point set of each frame skeleton graph provided by an embodiment of the present invention, and is specifically described as follows:

[0122] S301: Select a starting joint point from each joint point of each frame of the skeleton graph.

[0123] For each frame of the skeleton graph, a joint point is randomly selected from the joint points of the skeleton graph, and the selected joint point is used as the starting joint point. Figure 4 , which is a scene example diagram of the joint point direction cone feature of the skeleton graph provided in an embodiment of the present invention. The diagram contains 22 joint points, and the joint point numbered 1 is used as the starting joint point.

[0124] S302: For each frame of the skeleton image, the starting joint point of the skeleton image is matched with each remaining joint point in the skeleton image to obtain a plurality of second joint point groups.

[0125] For each frame of the skeleton graph, each second joint point group of the skeleton graph includes two joint points, one of which is a starting joint point.

[0126] Reference Figure 4 , match joint point 1 with each joint point of joint points 2-22 to obtain multiple second joint point groups, such as: (joint point 1, joint point 2), (joint point 1, joint point 3), (joint point 1, joint point 4), (joint point 1, joint point 19), which are not listed here.

[0127] S303: Determine the offset of each second joint point group of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image.

[0128] Continuing the description in S202, the skeleton diagram of the kth frame is used for description. Assuming that the second joint point group includes joint points and joint points Then the offset of the second joint point group can be expressed as: Wherein, i≠j, i and j can both represent the numbers of the joint points, and preferably, the value ranges are all positive integers.

[0129] S304: Based on the offsets of each frame of the skeleton image, the direction cone features of the joint points of each frame of the skeleton image are obtained.

[0130] Assuming that each joint point in each frame of the skeleton graph is numbered, and the joint point number is a positive integer starting from 1, preferably, the number of the selected starting joint point is 1, then the joint point direction cone feature of the skeleton graph of the kth frame can be expressed as: Among them, JDC k Represents the direction cone feature of the joint points in the k-th frame skeleton image, and N is the maximum number of joint points in the k-th frame skeleton image.

[0131] Reference Figure 4, which is a scene example diagram of the joint point direction cone feature of the skeleton graph provided in an embodiment of the present invention. As shown in the figure, the joint point numbered 1 is selected as the starting joint point, and the offset of each other joint point from the starting joint point is calculated, thereby obtaining the joint point direction cone feature of the skeleton graph.

[0132] S305: Determine an undirected graph of each frame skeleton graph.

[0133] It should be noted that the undirected graph is composed of the joints of the skeleton graph. It should be noted that the joints of the skeleton graph are connected, but the connection between nodes has no direction, thereby obtaining an undirected graph of the skeleton graph. Preferably, there is no closed loop connection between the joints in the undirected graph.

[0134] S306: For each frame of the skeleton image, the coordinates of each joint point of the skeleton image are used to process the undirected graph of the skeleton image to obtain the joint point direction tree features of the skeleton image.

[0135] The joint point direction tree is defined on the physical structure of the skeleton joint points. A joint point is selected on the hand skeleton as the root node of the joint point direction tree, and then the offsets between adjacent joint points are calculated step by step along the bones of the hand skeleton to obtain the joint point direction tree features of the skeleton graph.

[0136] Reference Figure 5 , which is a scene example diagram of the joint point direction tree feature of the skeleton graph provided in an embodiment of the present invention. The diagram contains multiple joint points, each of which has a corresponding number, numbered 1-22.

[0137] Reference Figure 6 , is a flow chart of a method for determining a direction tree feature of a joint point of a skeleton graph provided by an embodiment of the present invention, combined with Figure 5 To explain:

[0138] S601: Create a first node list, a second node list, and a feature list.

[0139] The first node list, the second node list and the feature list created at this time are all empty lists.

[0140] S602: Select a root node from each node of the undirected graph, and add the root node to the first node list and the second node list.

[0141] It should be noted that an undirected graph has an edge set and a node set, where the node set contains all the joints in the undirected graph, such as Figure 5 The joint points 1-22 in the edge set contain undirected edges connecting each joint point, such as Figure 5The undirected edges between joint point 1 and joint point 2 in the figure, the undirected edges between joint point 2 and joint points 3, 7, 11, 15, 19, etc., are not given here for illustration. Further, the undirected edges connecting joint point 1 and joint point 2 can be expressed as L 1-2 , further, because there is no direction in an undirected graph, L 2-1 It also represents the undirected edge connecting node 1 and node 2. Preferably, the undirected edge can also be represented by a node pair. For example, the undirected edge between node 1 and node 2 can be represented by a node pair <1,2>. 2-3 and L 3-2 Both represent the undirected edge connecting node 2 and node 3. Preferably, the node pair <2,3> can be used to represent the undirected edge between node 2 and node 3. 2-7 and L 7-2 Both represent an undirected edge connecting node 2 and node 7. Preferably, the joint point pair <2,7> can be used to represent the undirected edge between node 2 and node 7.

[0142] After selecting the root joint point, the root joint point is added to the preset node list. It should be noted that the selected root node is the root node of the joint point direction tree of the skeleton diagram.

[0143] For example, refer to Figure 5 , select joint point 1 as the root joint point, and add joint point 1 to the first node list.

[0144] S603: Determine whether the first node list is empty; if it is determined that the first node list is not empty, execute S604; if it is determined that the node list is empty, execute S609.

[0145] It should be noted that the first execution of S603 is performed after the root node is added. At this time, the first node list only contains the root node, so it should be determined that the node list is not empty.

[0146] When there are no more related nodes in the node list, it can be determined to be empty, which can indirectly indicate that each joint point in the undirected graph has been output as a target joint point.

[0147] S604: Output the target joint point from the first node list.

[0148] It should be noted that the target joint point output from the first node list for the first time is the root node added in S602.

[0149] Preferably, when this step is executed again later, if there are multiple joint points in the first joint point list, one joint point can be selected from each joint point at will as the target joint point to be output.

[0150] S605: Based on the second node list and the target joint point, determine whether there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph; if it is determined that there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph, execute S606; if it is determined that there is no undirected edge corresponding to the target joint point in the edge set of the undirected graph, return to execute S603.

[0151] Based on the second node list and the target joint point, the process of determining whether there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph is as follows:

[0152] Obtain each undirected edge containing the target joint point in the edge set of the undirected graph;

[0153] For each undirected edge, determine whether the joint point in the undirected edge that is not the target joint point is a joint point in the second node list; when it is determined that the joint point in the undirected edge that is not the target joint point is not a joint point in the second joint point list, determine that the undirected edge is an undirected edge corresponding to the target joint point; when it is determined that the joint point in the undirected edge that is not the target joint point is a joint point in the second joint point list, determine that the undirected edge is not an undirected edge corresponding to the target joint point.

[0154] For example, refer to Figure 5 , assuming that the target joint point output from the first node list is 1, and the second node list contains joint point 1, then the undirected edge containing joint point 1 is L 1-2 , because the joint point 2 in the undirected edge is not in the second joint point, we can determine that the undirected edge L 1-2 is the undirected edge corresponding to the target joint point; assuming that the target joint point output from the second joint point list is 2, and the second joint point list contains joint points 1 and 2, the undirected edge containing joint point 2 is L 2-3 , L 2-7 , L 2-11 , L 2-15 , L 2-19 , L 2-1 , because the undirected edge L 2-1 The joint point 1 in is the joint point in the second node list, so the undirected edge L 2-1 It is not an undirected edge corresponding to the target joint point, and the other undirected edges are all undirected edges corresponding to the target joint point.

[0155] S606: Determine each undirected edge corresponding to the target joint point as a target undirected edge.

[0156] Reference Figure 5 , assuming the target joint point is 1, because L 1-2 and L 2-1 They all represent the same edge, so L 1-2 (or L2-1 ) is determined as the target undirected edge.

[0157] S607: Based on the coordinates of the joint points of each target undirected edge, determine the vector of each target undirected edge, and add each vector to a preset feature list.

[0158] It should be noted that the vector of the target undirected edge is essentially the offset of the two joint points of the target undirected edge.

[0159] The vector operation method can be used to apply the coordinates of the two joint points of the target undirected edge to perform operations to obtain the vector of the target undirected edge.

[0160] S608: Based on the two joint points of each target undirected edge, update the first node list and the second node list, and return to execute S603.

[0161] It should be noted that, for each target undirected edge, the joint point that is not the target joint point among the two joint points is added to the first node list and the second node list, thereby completing the update of the first node list and the second joint point list.

[0162] It should be noted that after the first node list and the second node list are updated, when returning to execute S603, the first node list judged is the first node list updated in S608. Further, when executing S605 again, the second node list applied is the second node list updated in S608.

[0163] For example, the target joint point is 1 and the target undirected edge is L 2-1 , add joint point 2 to the first node list and the second node list.

[0164] S609: Based on each vector in the feature list, obtain the joint point direction tree feature of the skeleton graph.

[0165] Preferably, the feature list may be output as a joint point direction tree feature of the skeleton graph.

[0166] In the method provided by the embodiment of the present invention, after selecting the root joint point, the direction and offset between two joint points having a connection relationship are determined one by one, and then the joint point direction tree feature of the skeleton graph can be obtained.

[0167] For example, when determining the joint point direction tree features, a preset calculation module may be used to perform the following calculation process to obtain:

[0168] The input of the calculation module: G = (V, E) is an undirected graph, the vertex v1∈V is the root joint point of the joint point direction tree; V is a vertex set, including multiple joint points; E is an edge set, including multiple joint point pairs;

[0169] Output of the calculation module: JDT list, which is the joint point direction tree feature (JDT feature) of graph G;

[0170] Step 1: Initialize JDT to an empty list;

[0171] Step 2: Initialize the queue S to contain only the root node v1;

[0172] Step 3: Initialize the Visited set to a set containing only the root joint v1; the Visited here is different from the V in the above text.

[0173] Step 4: Check whether the queue S is empty. If the queue is not empty, go to step 5; if it is empty, go to step 9;

[0174] Step 5: Output a joint point from the queue S and record it as v2;

[0175] Step 6: Take the joint point pair corresponding to v2 from E (i.e., undirected edge), one of the joint points in the joint point pair corresponding to v2 is v2, and the other joint point is different from the joint points in the Visited set; convert the taken joint point pair into an offset and add it to the JDT list; for example, the joint point pair containing joint point v2<v2,v3> Can be uniformly converted to Among them, v3 is another joint point in the joint point pair, and the joint point to which v3 belongs in the joint point pair is added to the queue S and the set Visited;

[0176] Step 7: Return to step 4;

[0177] Step 8: End the algorithm and output the JDT list.

[0178] From the above content, it can be seen that the process of determining the node direction tree features of the skeleton graph does not construct a complex feature extraction module, and the calculation amount of the determination process is not large, thereby effectively reducing the calculation amount of the process of determining the gesture.

[0179] It should be noted that the method for obtaining the joint point direction tree features of the skeleton graph in the embodiment of the present invention is not limited to the above scheme, and the following scheme can also be used to obtain the joint point direction tree features of the skeleton graph, as described below:

[0180] S701: Select a root joint point from each joint point of the undirected graph, and add the root joint point to a preset first joint point set.

[0181] S702: Determine whether the first joint point set is empty; if it is determined that the first joint point set is not empty, execute S703; if it is determined that the first joint point set is empty, execute S708.

[0182] S703: Output the target joint point from the first node list.

[0183] S704: Determine whether there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph; if it is determined that there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph, execute S705; if it is determined that there is no undirected edge corresponding to the target joint point in the edge set of the undirected graph, execute S702.

[0184] It should be noted that the undirected edge corresponding to the target joint point is: an undirected edge that has not been marked as processed and contains the target joint point.

[0185] S705: Determine each undirected edge corresponding to the target joint point as a target undirected edge, and mark each undirected edge corresponding to the target joint point as processed.

[0186] S706: Based on the coordinates of the joint points of each target undirected edge, determine the vector of each target undirected edge, and add each vector to the feature list.

[0187] S707: Based on the two joint points of each target undirected edge, update the first joint point set, and return to execute S702.

[0188] When updating the first joint point set, the joint points in the target undirected edge that are not target joint points may be added to the first joint point set, thereby completing the update of the first joint point set.

[0189] When returning to execute S702 , the determined first joint point set is the first joint point set updated in S707 .

[0190] S708: Based on each vector in the feature list, obtain the joint point direction tree feature of the skeleton graph.

[0191] The method of obtaining the joint point direction tree features of the skeleton graph provided by the present invention does not require a large amount of calculation, effectively reducing the amount of calculation for gesture recognition, and can cover undirected graphs with rings and undirected graphs without rings, thereby expanding the application scenarios of the present invention.

[0192] S307: Based on the joint point direction cone features and the joint point direction tree features of each frame of the skeleton image, obtain the joint point set direction features of each frame of the skeleton image.

[0193] In the method provided by the embodiment of the present invention, the joint point set directional features of each frame of the skeleton image include node directional cone features and joint point directional tree features. Furthermore, in the process of extracting the joint point set directional features and joint point set distance features of each frame of the skeleton image, there is no need to apply a computationally intensive and complex neural network. The joint point set directional features and joint point set distance features of the skeleton image can be extracted based on the coordinates of the joint points. The computational complexity of the entire process is small, and the computational process is not complicated, which reduces the difficulty and complexity of gesture recognition and effectively lightweights the gesture recognition model.

[0194] S105: calling a preset feature extraction module to extract motion trajectory features of the skeleton sequence.

[0195] The feature extraction module includes a convolutional network, and the extracted motion estimation features are features of the time dimension, which can also be understood as features in the time domain; preferably, the motion trajectory features include features of the gesture during movement.

[0196] S106: Processing the motion trajectory features of the skeleton sequence, the distance features of the joint point sets of each frame of the skeleton image, and the direction features of the joint points to obtain a gesture recognition result of the skeleton sequence.

[0197] Based on the motion trajectory features of the skeleton sequence, the distance features of the joint point set of each frame skeleton image, and the direction features of the joint point, the gesture recognition result of the skeleton sequence can be output; preferably, the gesture recognition result includes the gesture recognized for the skeleton sequence, such as the gesture of fist, victory, etc.

[0198] In the method provided by the embodiment of the present invention, a preprocessed skeleton sequence is obtained; the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence are determined; based on the coordinates of each joint point of each frame of the skeleton image, the joint point set distance features and the joint point set direction features of each frame of the skeleton image are determined; a preset feature extraction module is called to extract the motion trajectory features of the skeleton sequence; the motion trajectory features of the skeleton sequence, the joint point distance features of each frame of the skeleton image, and the joint point set direction features are processed to obtain the gesture recognition result of the skeleton sequence. By using the coordinates of the joint points, the joint point set distance features and the joint point set method features of the skeleton image can be quickly extracted. The entire process does not require the construction of a complex and computationally intensive model, which reduces the complexity and computational complexity of gesture recognition, so that it can be applied to low-performance, low-power smart devices, reduce the cost of gesture recognition, and expand the application scenarios of gesture recognition.

[0199] refer to Figure 7 , is an execution flow chart of a gesture recognition model provided by an embodiment of the present invention, and is specifically described as follows:

[0200] The gesture recognition model receives a skeleton sequence, processes the skeleton sequence, obtains joint point set distance features and joint point set direction features of each frame of the skeleton sequence, and uses feature extraction model 701 to process the skeleton sequence to obtain motion trajectory features; processes the motion trajectory, joint point set distance features and joint point set direction features of each frame of the skeleton, and outputs the gesture recognition result of the skeleton sequence.

[0201] The convolution used in the gesture recognition model provided by the present invention is mainly 1D convolution, except when extracting motion trajectory features. We use the parameter filters to change the parameter amount of the model, thereby controlling the complexity of the model. According to the accuracy and computing power of the device, we generally choose filters of 16, 32 or 64. Figure 2 In the figure, the mark "CNN(1,2*filters), / 2" indicates that the convolution used is a 1D convolution (if it is a 2D convolution, it is represented by CNN2), the convolution kernel size is 1, the number of output channels is 2*filters, and the activation functions of BatchNorm and LeakyReLU are used. " / 2" means that the maximum pooling layer with a kernel size of 2 and a dropout with a probability of 0.1 on the channel will be used for the output of the convolution; the mark "⊕" in the figure means that the outputs of the three convolutions are connected in the channel dimension, and the whole process finally uses "Adaptive MaxPooling" to uniformly pool the time dimension of the output to ensure that the same fully connected layer is used when predicting the classification results; further Adaptive Max Pooling can fix the output size of the maximum pooling, and "FC(128)" means that the number of output channels of the fully connected layer is 128.

[0202] Preferably, the gesture recognition model needs to be trained before it is put into use, specifically: (1) setting the type of gesture to be recognized; (2) using an RGB camera to capture video, and then using the MediaPipe tool to extract the hand skeleton from the video; (3) using a public dataset combined with the captured skeleton image for training to establish a gesture recognition model; (4) applying the existing model to the gesture recognition device to recognize the captured gesture image.

[0203] During training, Adam (β1 = 0.9, β2 = 0.999) was selected as the optimizer, and ReduceLROnPlateau was used as the learning rate adjustment strategy to increase the learning rate from 10 -3 Down to 10 -5, using cross entropy as the loss function. The model code is implemented using PyTorch 1.2.0; first, the model parameters are randomly initialized and trained on public data to obtain the model weight parameters, and then the weight parameters are used as the initial value of the model to train the collected data set.

[0204] In practical applications, for dynamic gesture recognition, using only manually designed features makes it difficult for the system to learn more complex motion features, which in turn hinders the improvement of gesture recognition accuracy. Manually designed motion features often only describe the movement speed and acceleration of joints, which are features based on physical experience. However, for gesture recognition, how to effectively extract time domain features remains an open problem.

[0205] Graph convolution is difficult to extract the correlation features of two nodes on the graph when they are far apart, and graph convolution needs to consider the adjacency matrix and degree matrix representing the relationship between nodes during calculation, thus requiring more parameters and computational complexity. Therefore, the gesture recognition method in the prior art has a large amount of computation and requires high-performance and high-power computing equipment.

[0206] The gesture recognition method provided by the present invention uses manually designed computationally efficient and compact features in the spatial domain and automatically learned motion trajectory features in the temporal domain; in addition to directly using the coordinates of the skeleton joint points, the manually designed feature method also uses the connections between the joint points to design some geometric features. The main methods include:

[0207] (1) Three skeleton geometric features, namely the distance between bones, the angle between vectors of bone points, and the modulus ratio of vectors between bone points, are designed to describe gesture movements. Then, a multi-level gesture recognition model is established using the LSTM model.

[0208] (2) The global and local features of hand motion in space are proposed, and then the two features are fused to perform linear discriminant analysis (LDA) feature dimensionality reduction, and finally the Gaussian kernel support vector machine is used for dynamic gesture recognition and classification. Here, the global features of hand motion include the translation and rotation of the entire hand, and the local features are the translation and rotation of the fingers.

[0209] (3) The wrist joint and palm joint are used to describe the direction of the hand, the grouped joint coordinates are used to describe the shape of the hand, the motion information uses the velocity of the joint point as a feature, and finally a linear kernel SVM is used as a classifier to realize gesture recognition.

[0210] (4) It is proposed to calculate the Euclidean distance and offset of each joint point pair, as well as the dual velocity of the joint point motion trajectory as input features, and use a convolutional network to extract features to complete gesture classification. Although compared with the automatically learned features, the manually designed features are more compact and efficient, but because only a few fixed features are used, especially for the difficult-to-describe time domain features, the potential features of the skeleton sequence have yet to be fully explored.

[0211] Furthermore, the structure of the model provided by the present invention only uses convolutional networks with fewer parameters and computational complexity, especially one-dimensional convolution; the model provided by the present invention can be as small as 0.16MB in size with a minimum number of parameters and a computational complexity of 0.02GFLOPs; the results evaluated on the public data sets SHREC'17 and JHMDB are comparable to those of the best method, and the accuracy obtained is also high. Therefore, the gesture recognition method provided by the present invention is lightweight and efficient, and can be applied to low-power electronic devices, expand the application scenarios of gesture recognition, and effectively reduce the cost of gesture recognition.

[0212] and Figure 1 Corresponding to the method shown in FIG. 1 , an embodiment of the present invention further provides a gesture recognition device, the structural diagram of which is shown in FIG. Figure 8 The specific instructions are as follows:

[0213] An acquisition unit 801 is used to acquire a preprocessed skeleton sequence;

[0214] A first determining unit 802 is used to determine the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence;

[0215] A second determining unit 803 is used to determine the joint point set distance feature of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image;

[0216] The third determining unit 804 is used to determine the direction feature of the joint point set of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image;

[0217] A first extraction unit 805 is used to call a preset feature extraction module to extract motion trajectory features of the skeleton sequence;

[0218] The output unit 806 is used to process the motion trajectory features of the skeleton sequence, the joint point set distance features and the joint point direction features of the skeleton image of each frame to obtain the gesture recognition result of the skeleton sequence.

[0219] In the device provided by the embodiment of the present invention, a preprocessed skeleton sequence is obtained; the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence are determined; based on the coordinates of each joint point of each frame of the skeleton image, the joint point set distance features and the joint point set direction features of each frame of the skeleton image are determined; a preset feature extraction module is called to extract the motion trajectory features of the skeleton sequence; the motion trajectory features of the skeleton sequence, the joint point distance features of each frame of the skeleton image, and the joint point set direction features are processed to obtain the gesture recognition result of the skeleton sequence. By using the coordinates of the joint points, the joint point set distance features and the joint point set method features of the skeleton image can be quickly extracted. The whole process does not require the construction of a complex and computationally intensive model, which reduces the complexity and computational complexity of gesture recognition, so that it can be applied to low-performance, low-power intelligent devices, reduce the cost of gesture recognition, and expand the application scenarios of gesture recognition.

[0220] The device provided in the embodiment of the present invention may further include:

[0221] A collection unit, used for collecting videos to be identified;

[0222] A first extraction unit, used to extract an initial skeleton sequence from the video to be identified;

[0223] A denoising unit, used for performing denoising processing on the initial skeleton sequence to obtain a denoised skeleton sequence;

[0224] The normalization unit is used to normalize the denoised skeleton sequence to complete the preprocessing of the skeleton sequence.

[0225] In the device provided by the embodiment of the present invention, the second determining unit 803 may include:

[0226] A combining module, for each frame of the skeleton graph, combining each joint point in the skeleton graph in pairs to obtain a plurality of first joint point groups;

[0227] The first determination module is used to apply the coordinates of each joint point in the skeleton image of each frame to determine the Euclidean distance of each first joint point group of the skeleton image of the frame, and process each Euclidean distance to obtain the joint point set distance feature of the skeleton image of the frame.

[0228] In the device provided by the embodiment of the present invention, the third determining unit 804 may include:

[0229] A selection module, used for selecting a starting joint point from each joint point of the skeleton graph in each frame;

[0230] A matching module, for matching the starting joint point of the skeleton image with each remaining joint point in the skeleton image for each frame, to obtain a plurality of second joint point groups;

[0231] A second determination module is used to determine the offset of each second joint point group of the skeleton image of each frame based on the coordinates of each joint point of the skeleton image of each frame;

[0232] A first obtaining module is used to obtain the direction cone features of the joint points of the skeleton image of each frame based on the offsets of the skeleton image of each frame;

[0233] A third determination module is used to determine an undirected graph of the skeleton graph of each frame, wherein the undirected graph is composed of various joint points of the skeleton graph;

[0234] A second acquisition module is used for processing the undirected graph of the skeleton graph by applying the coordinates of each joint point of the skeleton graph for each frame of the skeleton graph to obtain the joint point direction tree features of the skeleton graph;

[0235] The third obtaining module is used to obtain the joint point set direction features of each frame of the skeleton image based on the joint point direction cone features and the joint point direction tree features of each frame of the skeleton image.

[0236] In the device provided by the embodiment of the present invention, the second obtaining module may include:

[0237] A creation subunit is used to create a first node list, a second node list and a feature list;

[0238] A selection subunit, configured to select a root node from among the nodes of the undirected graph, and add the root node to the first node list and the second node list;

[0239] A first judging subunit, used to judge whether the first node list is empty;

[0240] an output subunit, configured to output a target joint point from the first node list if it is determined that the first node list is not empty;

[0241] A second judgment subunit is used to judge whether there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph based on the second node list and the target joint point;

[0242] A first determining subunit is configured to determine each undirected edge corresponding to the target joint point as a target undirected edge if it is determined that there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph;

[0243] A second determining subunit is used to determine the vector of each target undirected edge based on the coordinates of the joint point of each target undirected edge, and add each vector to the feature list;

[0244] An updating subunit, used for updating the first node list and the second node list based on the two joint points of each of the target undirected edges, and triggering the first judging subunit to execute a step of judging whether the first node list is empty;

[0245] The obtaining subunit is used to obtain the joint point direction tree features of the skeleton graph based on each vector in the feature list if it is determined that the first node list is empty.

[0246] Furthermore, when the second judgment subunit determines that there is no undirected edge corresponding to the target joint point in the edge set of the undirected graph, the first judgment subunit is triggered to execute the step of determining whether the node list is empty.

[0247] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned gesture recognition method.

[0248] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Fig. 9 As shown, it specifically includes a memory 901 and one or more instructions 902, wherein the one or more instructions 902 are stored in the memory 901 and are configured to be executed by one or more processors 903 to execute the one or more instructions 902 and the gesture recognition method.

[0249] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0250] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0251] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0252] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture recognition method, characterized in that: Applicable to gesture recognition devices, including: Get the preprocessed skeleton sequence; Determine the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence; Based on the coordinates of each joint point of the skeleton image of each frame, determining the joint point set distance feature of the skeleton image of each frame; Based on the coordinates of each joint point of the skeleton image of each frame, determining the directional features of the joint point set of the skeleton image of each frame; Calling a preset feature extraction module to extract motion trajectory features of the skeleton sequence; Processing the motion trajectory features of the skeleton sequence, the distance features of the joint points of the skeleton image in each frame, and the direction features of the joint points to obtain a gesture recognition result of the skeleton sequence; Wherein, determining the joint point set distance feature of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image includes: For each frame of the skeleton graph, each joint point in the skeleton graph of the frame is combined in pairs to obtain a plurality of first joint point groups; For each frame of the skeleton image, the coordinates of each joint point in the skeleton image of the frame are applied to determine the Euclidean distance of each first joint point group of the skeleton image of the frame, and each of the Euclidean distances is processed to obtain the joint point set distance feature of the skeleton image of the frame; Wherein, determining the directional features of the joint point set of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image includes: Selecting a starting joint point from each joint point of the skeleton graph in each frame; For each frame of the skeleton image, matching the starting joint point of the skeleton image with each remaining joint point in the skeleton image to obtain a plurality of second joint point groups; Determine the offset of each second joint point group of the skeleton image of each frame based on the coordinates of each joint point of the skeleton image of each frame; Based on each offset of the skeleton image of each frame, a direction cone feature of a joint point of the skeleton image of each frame is obtained; Determine an undirected graph of the skeleton graph of each frame, wherein the undirected graph is composed of various joint points of the skeleton graph; For each frame of the skeleton graph, the coordinates of each joint point of the skeleton graph are applied to process the undirected graph of the skeleton graph to obtain a directional tree feature of the joint points of the skeleton graph; Based on the joint point direction cone features and joint point direction tree features of the skeleton image of each frame, the joint point set direction features of the skeleton image of each frame are obtained. Wherein, applying the coordinates of each joint point of the skeleton graph to process the undirected graph of the skeleton graph to obtain the joint point direction tree features of the skeleton graph includes: Creating a first node list, a second node list, and a feature list; Select a root node from each node of the undirected graph, and add the root node to the first node list and the second node list; Determine whether the first node list is empty; If it is determined that the first node list is not empty, outputting the target joint point from the first node list; Based on the second node list and the target joint point, determining whether there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph; If it is determined that there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph, each undirected edge corresponding to the target joint point is determined as a target undirected edge; Based on the coordinates of the joint points of each of the target undirected edges, determine the vector of each of the target undirected edges, and add each of the vectors to the feature list; Based on the two joint points of each of the target undirected edges, update the first node list and the second node list, and return to execute the step of determining whether the first node list is empty; If it is determined that the first node list is empty, then based on each vector in the feature list, a joint point direction tree feature of the skeleton graph is obtained.

2. The method according to claim 1, characterized in that The preprocessing process of the skeleton sequence includes: Collect the video to be identified; Extracting an initial skeleton sequence from the video to be identified; Performing denoising on the initial skeleton sequence to obtain a denoised skeleton sequence; The denoised skeleton sequence is normalized to complete the preprocessing of the skeleton sequence.

3. The method according to claim 1, characterized in that Also includes: If it is determined that there is no undirected edge corresponding to the target joint point in the edge set of the undirected graph, return to the step of determining whether the first node list is empty.

4. A gesture recognition device, characterized in that: include: An acquisition unit, used for acquiring a preprocessed skeleton sequence; A first determining unit, used to determine the coordinates of each joint point of each frame of the skeleton image in the skeleton sequence; A second determining unit is used to determine the joint point set distance feature of each frame of the skeleton image based on the coordinates of each joint point of the skeleton image of each frame; A third determining unit is used to determine the direction features of the joint point set of each frame of the skeleton image based on the coordinates of each joint point of each frame of the skeleton image; An extraction unit, used to call a preset feature extraction module to extract motion trajectory features of the skeleton sequence; An output unit, used to process the motion trajectory features of the skeleton sequence, the distance features of the joint points of the skeleton image in each frame, and the direction features of the joint points to obtain a gesture recognition result of the skeleton sequence; Wherein, the second determining unit is specifically used to: For each frame of the skeleton graph, each joint point in the skeleton graph of the frame is combined in pairs to obtain a plurality of first joint point groups; For each frame of the skeleton image, the coordinates of each joint point in the skeleton image of the frame are applied to determine the Euclidean distance of each first joint point group of the skeleton image of the frame, and each of the Euclidean distances is processed to obtain the joint point set distance feature of the skeleton image of the frame; Wherein, the third determining unit is specifically used for: Selecting a starting joint point from each joint point of the skeleton graph in each frame; For each frame of the skeleton image, matching the starting joint point of the skeleton image with each remaining joint point in the skeleton image to obtain a plurality of second joint point groups; Determine the offset of each second joint point group of the skeleton image of each frame based on the coordinates of each joint point of the skeleton image of each frame; Based on each offset of the skeleton image of each frame, a direction cone feature of a joint point of the skeleton image of each frame is obtained; Determine an undirected graph of the skeleton graph of each frame, wherein the undirected graph is composed of various joint points of the skeleton graph; For each frame of the skeleton graph, the coordinates of each joint point of the skeleton graph are applied to process the undirected graph of the skeleton graph to obtain a directional tree feature of the joint points of the skeleton graph; Based on the joint point direction cone features and joint point direction tree features of the skeleton image of each frame, the joint point set direction features of the skeleton image of each frame are obtained. Wherein, applying the coordinates of each joint point of the skeleton graph to process the undirected graph of the skeleton graph to obtain the joint point direction tree features of the skeleton graph includes: Creating a first node list, a second node list, and a feature list; Select a root node from each node of the undirected graph, and add the root node to the first node list and the second node list; Determine whether the first node list is empty; If it is determined that the first node list is not empty, outputting the target joint point from the first node list; Based on the second node list and the target joint point, determining whether there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph; If it is determined that there is an undirected edge corresponding to the target joint point in the edge set of the undirected graph, each undirected edge corresponding to the target joint point is determined as a target undirected edge; Based on the coordinates of the joint points of each of the target undirected edges, determine the vector of each of the target undirected edges, and add each of the vectors to the feature list; Based on the two joint points of each of the target undirected edges, update the first node list and the second node list, and return to execute the step of determining whether the first node list is empty; If it is determined that the first node list is empty, then based on each vector in the feature list, a joint point direction tree feature of the skeleton graph is obtained.

5. The device according to claim 4, characterized in that Also includes: A collection unit, used for collecting videos to be identified; A first extraction unit, used to extract an initial skeleton sequence from the video to be identified; A denoising unit, used for performing denoising processing on the initial skeleton sequence to obtain a denoised skeleton sequence; The normalization unit is used to normalize the denoised skeleton sequence to complete the preprocessing of the skeleton sequence.

6. A storage medium, characterized in that: The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the gesture recognition method according to any one of claims 1 to 3.

7. An electronic device, characterized in that: The device comprises a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to perform the gesture recognition method according to any one of claims 1 to 3.

Images