A three-dimensional dynamic hand gesture recognition method based on improved dynamic time warping algorithm

Abstract

The present invention relates to a three-dimensional dynamic gesture recognition method based on an improved dynamic time warping algorithm, belonging to the field of robot vision. The hardware system of the method includes a left camera, a right camera and a host computer that are parallel to each other; the specific recognition operation steps are as follows: (1) static gesture recognition, selecting the static gesture "one" of extending one finger as the recognition stop instruction, and the static gesture "five" of extending five fingers as the recognition start instruction; (2) eliminating redundant movements; (3) obtaining the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node; (4) completing the three-dimensional dynamic gesture recognition based on the established three-dimensional dynamic gesture classification template library. The three-dimensional dynamic gesture recognition method of the present invention enables the three-dimensional dynamic gesture recognition system to have a higher recognition accuracy rate for complex three-dimensional dynamic gestures.

Description

Technical Field

[0001] The present invention belongs to the field of robot vision, and in particular relates to a high-accuracy three-dimensional dynamic gesture recognition method. Background Art

[0002] Gesture recognition is an emerging form of human-computer interaction that offers advantages over traditional methods, such as being natural, intuitive, and easy to learn. Currently, gesture recognition is implemented through wearable devices and visual sensors. Wearable gesture recognition requires expensive external equipment and is inconvenient to use, so vision-based gesture recognition is the current mainstream research direction. Vision-based gesture recognition is divided into static gesture recognition and dynamic gesture recognition. Static gesture recognition only identifies the user's gesture state in the current image. While easier to implement than dynamic gesture recognition, it contains less gesture information and can recognize fewer gestures. Dynamic gesture recognition, on the other hand, recognizes the user's entire dynamic movement over a period of time. While more challenging to implement than static gesture recognition, it offers a wider range of gestures and, therefore, offers greater application prospects than static gesture recognition.

[0003] Vision-based dynamic gesture recognition can be achieved through deep learning methods or traditional matching algorithms. Deep learning methods require training the recognition model using a large training set containing large amounts of data, which consumes a large amount of computation. Furthermore, the trained model can only recognize the gesture types included in the training set. Adding new gesture types requires re-creating the training set and retraining the model, making it inconvenient. Compared to deep learning methods, first acquiring gesture trajectory sequences and then using traditional matching algorithms to perform trajectory matching is more convenient. Adding new gestures requires simply adding a new template trajectory sequence to the gesture template library, and the system consumes less computational resources during runtime. Traditional matching algorithms currently used for trajectory matching include hidden Markov chains, support vector machines, and dynamic time warping. However, hidden Markov chains are limited by the homogeneous Markov assumption and the observation independence assumption, which dynamic gesture trajectories cannot guarantee. Therefore, hidden Markov chains perform poorly in dynamic gesture recognition. The support vector machine algorithm can achieve similarity matching between two trajectory sequences of equal length. However, due to fluctuations in the speed of the subject's movements, the resulting trajectory sequences are often of unequal length. To utilize the support vector machine algorithm, the sequences must be forcibly scaled, which results in a loss of motion details and affects accuracy. The dynamic time warping algorithm accounts for the effects of speed fluctuations by performing one-to-many and many-to-one matching of the points in the two sequences to achieve a local scaling effect. It then recursively finds the matching method that minimizes the cumulative spatial distance between matching points, achieving the optimal match. Therefore, the DTW algorithm is well-suited for similarity matching between gesture trajectory sequences of varying lengths. However, traditional dynamic time warping algorithms only consider the matching relationship between the points in the two sequences and fail to consider the impact of the total sequence length on the cumulative distance between matching points. In fact, the longer the total sequence length, the larger the cumulative distance between matching points. Therefore, using this cumulative distance as the basis for gesture classification can interfere with recognition results. It is necessary to improve the way the traditional dynamic time warping algorithm accumulates matching point distances to eliminate the influence of total sequence length on similarity matching results.

[0004] When classifying dynamic gestures, the quality of the template library, which serves as the classification standard, significantly impacts recognition accuracy. To improve the accuracy of dynamic gesture recognition, the accuracy of the template trajectory sequences in the template library needs to be enhanced. Currently, the mainstream approach to improving template library quality is to record multiple template trajectory sequences for each action and then compare the matching results of each template trajectory sequence with the actual test trajectory sequence to identify the best template trajectory sequence in the template library. Alternatively, during dynamic gesture recognition, the trajectory sequence to be tested is matched against all template trajectory sequences in the template library for that action, and the match with the highest similarity is selected as the dynamic gesture recognition result. Because the trajectory sequence recorded by the system is susceptible to interference, and testers may not be able to repeat the same action completely accurately, the template trajectory sequence inevitably contains irregular and gross errors. The more complex the action, the greater the gross errors in the trajectory sequence. Current template library optimization methods simply select a template trajectory sequence with a smaller error from multiple template trajectory sequences with errors. Therefore, they are still subject to the irregular and gross errors generated during the recording of a single trajectory sequence. Therefore, a method is needed to synthesize multiple template trajectory sequences into a refined template trajectory sequence so that the random errors of multiple template trajectory sequences offset each other and eliminate the influence of irregular gross errors generated by single trajectory sequence recording on the recognition results. Summary of the Invention

[0005] To achieve high-accuracy and high-stability three-dimensional dynamic gesture recognition, while ensuring a fast system response speed without excessively increasing the amount of computation, and improving the recognition accuracy of the dynamic gesture system for various dynamic gestures, especially complex dynamic gestures, the present invention provides a three-dimensional dynamic gesture recognition method and system based on an improved DTW algorithm and a template synthesis algorithm.

[0006] A hardware system for a dynamic gesture recognition method based on an improved DTW algorithm includes a left camera, a right camera, and a host computer. The left and right cameras form a binocular depth camera, and the binocular depth camera is connected to the host computer; the host computer is also called a host computer; the host computer is equipped with a visual processing unit (GPU) that supports deep learning.

[0007] The specific steps to achieve complete gesture recognition based on the improved DTW algorithm are as follows:

[0008] (1) Static gesture recognition of left-hand gestures

[0009] (1.1) Gesture segmentation

[0010] The left and right cameras of the binocular depth camera simultaneously shoot and transmit the captured left and right images to a host computer. The host computer, based on the principle of imitating the human eye's judgment of object distance, compares the differences between the left and right images and calculates the three-dimensional coordinates of each point in the left image relative to the midpoint of the line connecting the left and right cameras, thereby obtaining a three-dimensional point cloud containing three-dimensional coordinate information. Based on the three-dimensional point cloud, when a human body appears in the left and right images, a three-dimensional point cloud of the human body is obtained. The visual processor performs deep learning on the three-dimensional point cloud of the human body to realize human skeleton recognition, obtains the three-dimensional coordinates of the 18 joint nodes of the human body in the left image and the pixel coordinates of the 18 joint nodes in the left image, and thus obtains a skeleton recognition model.

[0011] From the skeleton recognition model, the left wrist node is selected as the basis for locating the left hand position, and the three-dimensional coordinates of the left wrist node and the pixel coordinates in the left image are obtained; the left palm image within a rectangular area above the pixel coordinates of the left wrist node in the left image is intercepted, and the left palm image is segmented from the left image to obtain a left palm image containing only the left palm portion, completing the gesture segmentation of static gesture recognition;

[0012] (1.2) Image preprocessing

[0013] The host computer converts the color gamut of the left palm image from the three primary colors of red, green and blue (RGB) to hue-saturation value (HSV), and considers pixels with hue-saturation values of H<=19, S>=48, and V>40 as skin pixels, and pixels that do not meet the requirements of skin pixels as background pixels, thereby binarizing the left palm image to distinguish between skin pixels and background pixels; then, morphological dilation is performed on the binarized left palm image to eliminate skin pixels in the palm that are mistakenly identified as background pixels; then, median filtering is performed to eliminate jagged contours on the palm edge and smooth the edges; finally, contour extraction is performed to eliminate skin pixels and background pixels outside the palm edge contour, thereby obtaining a complete and clear left palm contour image, thereby completing image preprocessing;

[0014] (1.3) Static gesture classification

[0015] The host computer detects multiple inward-facing concave structures on the left palm outline image. These inward-facing concave structures are called convex hull defects. The number of large convex hull defects among all convex hull defects is recorded. The number of large convex hull defects plus one is the number of extended fingers in the left palm outline. Static gestures are classified according to the number of extended fingers.

[0016] A large convex hull defect is a convex hull defect with a depth exceeding one-quarter of the length of the palm.

[0017] The static gesture "one" of extending one finger is selected as the stop instruction of dynamic gesture recognition, and the static gesture "five" of extending five fingers is selected as the start instruction of dynamic gesture recognition;

[0018] (2) Eliminate redundant actions

[0019] Before starting 3D dynamic gesture recognition, first start the static gesture recognition of the left hand. The static gesture "five" of extending five fingers is used as the starting instruction for 3D dynamic gesture recognition. After the tester makes the "five" gesture with his left hand, the 3D dynamic gesture recognition officially begins.

[0020] The static gesture "1" of extending one finger is used as the stop instruction of 3D dynamic gesture recognition. The 3D dynamic gesture recognition is stopped after the tester makes the gesture "1" with his left hand;

[0021] The dynamic action between the two static gestures is regarded as a 3D dynamic gesture that needs to be recognized, and the redundant dynamic gestures that are not desired to be recognized are ignored;

[0022] (3) Obtain the three-dimensional motion trajectory sequence of the right wrist node and the right elbow node

[0023] When receiving the 3D dynamic gesture recognition start command, the host computer continuously receives the left and right video images sent back by the binocular depth camera, and continuously identifies and tracks the left shoulder node, right shoulder node, right elbow node, and right wrist node in the left video image through the skeleton recognition model in step (1.1), and obtains four 3D original trajectory sequences of the left shoulder node, right shoulder node, right elbow node, and right wrist node. The four 3D original trajectory sequences are preprocessed to obtain the 3D motion trajectory sequences of the right wrist node and right elbow node. The preprocessing steps are as follows:

[0024] (3.1) Obtaining the original three-dimensional trajectory sequence of joint points

[0025] The left and right cameras of the binocular depth camera continuously shoot simultaneously and transmit the left and right video images back to the host computer. The host computer uses the skeleton recognition model in step (1.1) to identify and track the four nodes in the left video image: the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node. A total of 200 consecutive frames of images are selected as subsequent processing images;

[0026] Obtain four three-dimensional coordinates in each of the 200 frames of images, which are the three-dimensional coordinates of the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node relative to the midpoint of the line connecting the left camera and the right camera, respectively. One three-dimensional coordinate represents a sequence point, and 200 sequence points of each of the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node are obtained respectively; the 200 sequence points of the left shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain a three-dimensional original trajectory sequence of the left shoulder node; the 200 sequence points of the right shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain a three-dimensional original trajectory sequence of the right shoulder node; the 200 sequence points of the right elbow shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain a three-dimensional original trajectory sequence of the right elbow node; the 200 sequence points of the right wrist node are arranged from front to back according to the time sequence in the 200 frames of images to obtain a three-dimensional original trajectory sequence of the right wrist node, and a total of four three-dimensional original trajectory sequences are obtained;

[0027] (3.2) Obtain the three-dimensional relative trajectory sequence of the right wrist node and the right elbow node

[0028] Set a variable q with an initial value of 1, take the horizontal projection of the line connecting the qth sequence point in the three-dimensional original trajectory sequence of the left shoulder node and the qth sequence point in the three-dimensional original trajectory sequence of the right shoulder node as the x-axis, set the vertical direction as the z-axis, and construct a right-handed Cartesian coordinate system as the qth human coordinate system according to the right-hand rule of the Cartesian coordinate system; calculate the relative coordinates of the qth sequence point in the three-dimensional original trajectory sequence of the right wrist node in the qth human coordinate system through the coordinate transformation formula, representing the qth sequence point in the three-dimensional relative trajectory sequence of the right wrist node; calculate the relative coordinates of the qth sequence point in the three-dimensional original trajectory sequence of the right elbow node in the qth human coordinate system through the coordinate transformation formula, representing the qth sequence point in the three-dimensional relative trajectory sequence of the right elbow node;

[0029] Keep increasing the q value by one and repeat step (3.2) until the q value reaches 200, and obtain the three-dimensional relative trajectory sequence of the right wrist node and the three-dimensional relative trajectory sequence of the right elbow node respectively;

[0030] (3.3) Obtain the three-dimensional motion trajectory sequence of the right wrist node and the right elbow node respectively

[0031] Calculate the wrist distance between adjacent sequence points in the three-dimensional relative trajectory sequence of the right wrist node, and calculate the elbow distance between adjacent sequence points in the three-dimensional relative trajectory sequence of the right elbow node;

[0032] The motion speed of each sequence point in the three-dimensional relative trajectory sequence of the right wrist node is calculated according to the wrist distance, and each sequence point in the three-dimensional relative trajectory sequence of the right wrist node is classified into high-speed points and low-speed points based on the motion speed;

[0033] The motion speed of each sequence point in the three-dimensional relative trajectory sequence of the right elbow node is calculated according to the elbow distance, and each sequence point in the three-dimensional relative trajectory sequence of the right elbow node is classified into high-speed points and low-speed points based on the motion speed;

[0034] The continuously distributed low-speed points are set as the static phase, and the continuously distributed high-speed points are set as the moving phase. After eliminating the static phase, the three-dimensional motion trajectory sequence of the right wrist node containing only the moving phase and the three-dimensional motion trajectory sequence of the right elbow node containing only the moving phase are obtained respectively;

[0035] The continuous distribution means that: more than five consecutive points in the three-dimensional relative trajectory sequence of the right elbow node are simultaneously classified as low-speed points or high-speed points; or more than five consecutive points in the three-dimensional relative trajectory sequence of the right wrist node are simultaneously classified as low-speed points or high-speed points;

[0036] (4) 3D dynamic gesture recognition

[0037] According to the 3D dynamic gesture classification template library, the specific steps for dynamic gesture recognition are as follows:

[0038] (4.1) Obtaining the most similar dynamic gesture

[0039] The tester makes a 3D dynamic gesture to be recognized. According to steps (2) and (3), the motion trajectory sequence of the right wrist node and the motion trajectory sequence of the right elbow node are obtained. Then, the motion trajectory sequences of the two joint points are matched with each 3D dynamic gesture classification template in the 3D dynamic gesture classification template library by using the improved dynamic time warping algorithm. The specific steps of similarity matching are as follows:

[0040] Calculate the improved dynamic time warping distance between the motion trajectory sequence of the right wrist node to be identified and the fine template trajectory sequence of the right wrist node in the three-dimensional dynamic gesture classification template to obtain the improved dynamic time warping distance of the right wrist node;

[0041] Calculate the improved dynamic time warping distance between the motion trajectory sequence of the right elbow node to be identified and the fine template trajectory sequence of the right elbow node in the three-dimensional dynamic gesture classification template to obtain the improved dynamic time warping distance of the right elbow node;

[0042] The average value of the improved dynamic time warping distance of the right wrist node and the improved dynamic time warping distance of the right elbow node is taken as the similarity evaluation value between the motion trajectory sequence of the right wrist node or right elbow node to be identified and the 3D dynamic gesture classification template. The smaller the similarity evaluation value, the more similar it is.

[0043] A 3D dynamic gesture classification template that is most similar to the motion trajectory sequence of the right wrist node or right elbow node to be identified is selected from the 3D dynamic gesture classification template library. The 3D dynamic gesture corresponding to the 3D dynamic gesture classification template is recorded as the most similar dynamic gesture, and the corresponding similarity evaluation value is called the minimum dynamic time warping distance.

[0044] The improved dynamic time warping algorithm is obtained by adding a correction operation step to the dynamic time warping algorithm;

[0045] (4.2) Obtaining the results of three-dimensional dynamic gesture recognition

[0046] If the minimum dynamic time warping distance is less than 0.2m, the 3D dynamic gesture to be recognized is considered to be successfully recognized, and the most similar dynamic gesture is the gesture recognition result;

[0047] If the minimum dynamic time warping distance value is greater than 0.2m, it is considered that the 3D dynamic gesture to be identified by the tester and the most similar dynamic gesture do not belong to the same dynamic gesture, that is, the dynamic gesture made by the tester has not yet been recorded in the 3D dynamic gesture classification template library. The tester can choose to start making a classification template for the 3D dynamic gesture to be identified and add the classification template to the template library, or ignore the 3D dynamic gesture to be identified and directly start the next 3D dynamic gesture recognition. The operation of establishing the 3D dynamic gesture classification template library is as follows:

[0048] Before officially starting the gesture recognition operation, the tester needs to repeat the target three-dimensional dynamic gesture continuously, and continuously extract the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node through steps (2) and (3). The first extracted three-dimensional motion trajectory of the right wrist node is set as the rough template trajectory sequence of the right wrist node, and the first extracted three-dimensional motion trajectory sequence of the right elbow node is set as the rough template trajectory sequence of the right elbow node; in the process of the tester continuously repeating the target three-dimensional dynamic gesture, the upper computer continuously synthesizes and replaces the two rough template trajectory sequences through the template synthesis algorithm, and completes 200 template synthesis and replacements of the right hand. The rough template trajectory sequence of the wrist node is called the fine template trajectory sequence of the right wrist node, and the rough template trajectory sequence of the right elbow node after 200 template synthesis and replacement is called the fine template trajectory sequence of the right elbow node. The two fine template trajectory sequences are collectively called the three-dimensional dynamic gesture classification template of the target three-dimensional dynamic gesture; new target three-dimensional dynamic gestures are continuously designed and three-dimensional dynamic gesture classification templates are obtained until the number of target three-dimensional dynamic gestures reaches four or more, that is, more than four three-dimensional dynamic gesture classification templates are obtained. More than four three-dimensional dynamic gesture classification templates constitute a three-dimensional dynamic gesture classification template library, which is used as the basis for gesture recognition when formally performing gesture recognition operations.

[0049] The identification technology solutions are further defined as follows:

[0050] The distance between the lens of the left camera and the lens of the right camera is 120 mm;

[0051] The field of view of the binocular depth camera is 110° horizontally × 70° vertically × 120° diagonally.

[0052] In step (1.3), the large convex hull defect refers to a convex hull defect with a concave depth exceeding one-quarter of the length of the palm.

[0053] In step (3.2), the coordinate transformation formula is a formula for calculating the three-dimensional relative coordinates of the qth sequence point in the three-dimensional relative trajectory sequence of the right elbow node or the qth sequence point in the three-dimensional relative trajectory sequence of the right wrist node relative to the qth human body relative coordinate system. The specific formula is as follows:

[0054]

[0055]

[0056]

[0057]

[0058] In formula (1)(2)(3)(4), (x A,y A ,z A ) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the left shoulder node, (x B ,y B ,z B ) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the right shoulder node, (x, y, z) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the right elbow node or the qth sequence point in the three-dimensional original trajectory sequence of the right wrist node that requires coordinate transformation, (x′, y′, z′) is the three-dimensional relative coordinate of the qth sequence point in the three-dimensional relative trajectory sequence of the right elbow node or the qth sequence point in the three-dimensional relative trajectory sequence of the right wrist node that requires relative coordinate calculation in the qth human body relative coordinate system, and the unit of all coordinates is m.

[0059] In step (3.3), all sequence points in the three-dimensional relative trajectory sequence are divided into two types: low-speed points and high-speed points through a clustering algorithm. The specific operation steps are as follows:

[0060] (3.3.1) Calculate the velocity of all sequence points in the three-dimensional relative trajectory sequence. Randomly select two sequence points a and b in the three-dimensional relative trajectory sequence. Calculate the difference between the velocity of all sequence points in the entire sequence trajectory and the velocity of sequence points a and b. If the velocity difference between a sequence point and sequence point a is less than the velocity difference between a sequence point and sequence point b, then classify the sequence point as Class A. If the velocity difference between a sequence point and sequence point a is greater than the velocity difference between a sequence point and sequence point b, then classify the sequence point as Class B.

[0061] (3.3.2) Calculate the average speed of A for class A and the average speed of B for class B, find the sequence point in class A whose speed is closest to the average speed of A, and use this sequence point to replace the original sequence point a; find the sequence point in class B whose speed is closest to the average speed of B, and use this sequence point to replace the original sequence point b; repeat step (3.3.1) once to reclassify all sequence points;

[0062] (3.3.3) Repeat step (3.3.2) at least twice, and check whether sequence point a and sequence point b are replaced each time. If sequence point a and sequence point b are no longer replaced with the repetition of step (3.3.2), it means that the optimal classification has been achieved at this time. Stop repeating step (3.3.2) and regard the class with the larger average speed between class A and class B as the high-speed class, and all sequence points in the high-speed class as high-speed points. The class with the smaller average speed is regarded as the low-speed class, and all sequence points in the low-speed class as low-speed points.

[0063] In step (4), the steps for establishing the three-dimensional dynamic gesture classification template library are as follows:

[0064] (A) Obtaining transition trajectory sequence

[0065] Each time the tester repeats the target three-dimensional dynamic gesture, the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node are extracted through steps (2) and (3), and the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node are respectively matched with the recorded rough template trajectory sequences of the two joint points through the improved dynamic time warping algorithm to obtain the improved dynamic time warping distance of the right wrist node between the three-dimensional motion trajectory sequence of the right wrist node and the rough template trajectory sequence of the right wrist node, and the improved dynamic time warping distance of the right elbow node between the three-dimensional motion trajectory sequence of the right elbow node and the rough template trajectory sequence of the right elbow node, thus completing the similarity calculation;

[0066] The improved dynamic time warping distance of the right wrist node and the improved dynamic time warping distance of the right elbow node respectively represent the similarity between the three-dimensional motion trajectory sequences of the two joint points and the rough template trajectory sequence. If the obtained improved dynamic time warping distances of the right wrist node or the right elbow node are both less than 0.2m, it is considered that the three-dimensional motion trajectory sequences of the two joint points are sufficiently similar to the recorded rough template trajectory sequence; the three-dimensional motion trajectory sequences of the two joint points are called the transition trajectory sequences of the two joint points, that is, the transition trajectory sequence of the right wrist node and the transition trajectory sequence of the right elbow node;

[0067] If one of the obtained improved dynamic time warping distances of the right wrist node and the improved dynamic time warping distances of the right elbow node is greater than 0.2m or both are greater than 0.2m, repeat the similarity calculation until both obtained improved dynamic time warping distances are less than 0.2m. The three-dimensional motion trajectory sequences of the two joints are called the transition trajectory sequences of the two joints, that is, the transition trajectory sequence of the right wrist node and the transition trajectory sequence of the right elbow node.

[0068] (B) Obtaining a fine template trajectory sequence

[0069] The transition trajectory sequence of the right wrist node and the rough template trajectory sequence of the right wrist node are synthesized by the template synthesis algorithm to obtain a new rough template trajectory sequence of the right wrist; the transition trajectory sequence of the right elbow node and the rough template trajectory sequence of the right elbow node are synthesized to obtain a new rough template trajectory sequence of the right elbow; and the previous rough template trajectory sequences are replaced and synthesized again. After accumulating 200 synthesis and replacement, the fine template trajectory sequences of the right wrist node and the fine template trajectory sequences of the right elbow node are obtained respectively.

[0070] (C) Obtaining a 3D dynamic gesture classification template library

[0071] The fine template trajectory sequence of the right wrist node and the fine template trajectory sequence of the right elbow node are recorded as a set of fine template trajectory sequences, which are called the first 3D dynamic gesture classification template. The host computer stores the first 3D dynamic gesture classification template in a blank file, which is called the 3D dynamic gesture classification template library.

[0072] Assume that an unrecorded 3D dynamic gesture is used as the second target 3D dynamic gesture. A second 3D dynamic gesture classification template of the second 3D dynamic gesture is obtained through steps (A) and (B), and stored in the 3D dynamic gesture classification template library. New target 3D dynamic gestures are repeatedly designed, and several 3D dynamic gesture classification templates are obtained and stored in the 3D dynamic gesture classification template library until the number of recorded 3D dynamic gestures in the 3D dynamic gesture classification template library is greater than 4, thereby obtaining a 3D dynamic gesture classification template library that can be used as a basis for 3D dynamic gesture classification.

[0073] The technical solution for establishing a further limited three-dimensional dynamic gesture classification template library is as follows:

[0074] In step (A), the steps for performing similarity matching using the improved dynamic time warping algorithm are as follows:

[0075] The right elbow node similarity matching operation is as follows:

[0076] (A1) Construct an m×n matrix, called a distance matrix, where the element in the i-th row and j-th column of the distance matrix is the Euclidean distance between the i-th sequence point in the three-dimensional motion trajectory sequence of the right elbow node and the j-th sequence point in the rough template trajectory sequence of the right elbow node;

[0077] The m in the distance matrix is the number of sequence points contained in the three-dimensional motion trajectory sequence of the right elbow node;

[0078] n in the distance matrix is the number of sequence points contained in the rough template trajectory sequence of the right elbow node;

[0079] (A2) Then construct an m×n matrix, called the cumulative distance matrix. The element in the i-th row and j-th column of the cumulative distance matrix is obtained by the recursive formula (5):

[0080] D ij =d ij +min{D i,j-1 ,D i-1,j-1 ,D i-1,j}——(5)

[0081] In formula (5), d ij is the element in row i and column j of the distance matrix, D ij To accumulate the elements in row i and column j of the distance matrix, when row number i or column number j is less than 1, 0 is used instead of D ij , D11 =d 11 , starting from the element in the first row and first column, calculate all the elements in the cumulative distance matrix from the first column to the last column by recursive formula (5), then the element D in the mth row and nth column is mn The dynamic time warping distance between two trajectory sequences calculated by the traditional dynamic time warping algorithm will increase with the increase of m and n, so it needs to be corrected;

[0082] (A3) The correction steps are as follows:

[0083] In order to correct the influence of the size of m and n on the traditional dynamic time warping distance, from D mn First, recursively follow the recursive formula (5) to find and record all the distance matrices in step (A1) that participated in D mn Calculate the element d ij , these elements d ij The set of is called the minimum warping path between the two trajectory sequences; the elements in the minimum warping path in each column of the distance matrix are averaged, the average values of each column are accumulated and divided by the total number of columns n to obtain the improved dynamic time warping distance between the three-dimensional motion trajectory sequence of the right elbow node and the rough template trajectory sequence of the right elbow node; the smaller the improved dynamic time warping distance, the higher the similarity;

[0084] The size of the improved dynamic time warping distance is not affected by the sizes of m and n like the traditional dynamic time warping distance.

[0085] The similarity matching operation of the right wrist node is the same as that of the right elbow node;

[0086] That is, the improved dynamic time warping distance of the right elbow node and the improved dynamic time warping distance of the right wrist node are obtained;

[0087] If both improved dynamic time warping distances are less than 0.2m, the new 3D motion trajectory sequences of the two joints are considered sufficiently similar to the recorded rough template trajectory sequence. If one or both of the two improved dynamic time warping distances are greater than 0.2m, repeat step (A) until both improved dynamic time warping distances are less than 0.2m; thus, the transition trajectory sequences of the right elbow node and the right wrist node are obtained respectively.

[0088] In step (B), the template synthesis and replacement operations are performed 200 times cumulatively using the template synthesis algorithm as follows:

[0089] The template synthesis and replacement operations of the right elbow node are as follows:

[0090] (B1) Obtain two weight values

[0091] The template is synthesized by the rough template trajectory sequence of the right elbow node and the transition trajectory sequence of the right elbow node. The weight value of the rough template trajectory sequence of the right elbow node is set to e, and the weight value of the transition trajectory sequence of the right elbow node is set to f.

[0092] The 3D motion trajectory sequence of the right elbow node recorded when the tester first performs the target 3D dynamic gesture is used as the initial rough template trajectory sequence of the right elbow node. The weight value e is recorded as 1. After that, the weight value e of the rough template trajectory sequence of the right elbow node is increased by 1 each time the template synthesis is performed.

[0093] The weight value f of the right elbow node transition trajectory sequence in template synthesis is always 1;

[0094] (B2) Obtaining the incremental sequence points

[0095] Repeat step (A) once to obtain a right elbow node transition trajectory sequence, and record the minimum regularized path between the right elbow node transition trajectory sequence generated in the correction operation of step (A3) and the right elbow node rough template trajectory sequence;

[0096] Assume that the number of Euclidean distance values contained in the minimum regularized path is u; each of the u Euclidean distance values is obtained by calculating the Euclidean distance between a sequence point in the transition trajectory sequence of the right elbow node and a sequence point that matches the rough template trajectory sequence of the right elbow node. The two sequence points used to calculate each Euclidean distance value are recorded as a matching point pair for each Euclidean distance value, and the u Euclidean distance values correspond to u matching point pairs respectively.

[0097] u Euclidean distance values are respectively located in the distance matrix of step (A1), and the row number and column number of each Euclidean distance value in the distance matrix are recorded. The row number of each Euclidean distance value in the distance matrix is set as the row value of the matching point pair of each Euclidean distance value, and the column number of each Euclidean distance value in the distance matrix is set as the column value of the matching point pair of each Euclidean distance value;

[0098] For each of the u matching point pairs, calculate the weighted average coordinates of each matching point pair based on the two three-dimensional coordinates of each matching point pair according to the weight value e and the weight value f recorded in step (B1), to obtain u weighted average coordinates; set each weighted average coordinate as the three-dimensional coordinate of an incremental point sequence point, to obtain u incremental point sequence points; according to the correspondence between the u incremental point sequence points and the u matching points, set the row value of each matching point pair as the row value of the corresponding incremental point sequence point, and set the column value of each matching point pair as the column value of the corresponding incremental point sequence point, to obtain u incremental point sequence points with row values and column values;

[0099] (B3) Single-point template synthesis and replacement

[0100] Sort u increment sequence points with row values and column values, arranging increment sequence points with smaller row values before increment sequence points with larger row values, and arranging increment sequence points with smaller column values before increment sequence points with larger column values, to obtain sorted u increment sequence points; all sorted u increment sequence points together constitute a trajectory sequence, which is set as the increment trajectory sequence;

[0101] Delete the right elbow node rough template trajectory sequence, set the added point trajectory sequence to the right elbow node rough template trajectory sequence with the number of synthesis increased by one, complete one point-added template synthesis and replacement, and obtain the right elbow node rough template trajectory sequence with the number of times of participation in synthesis increased by one, that is, the right elbow node rough template trajectory sequence with the weight value e increased by one;

[0102] (B4) Complete 100 times of template synthesis

[0103] Repeat steps (B1), (B2), and (B3) to complete one hundred times of template synthesis and replacement, and obtain a rough template trajectory sequence of the right elbow node with one hundred synthesis times, and the weight value e reaches 101;

[0104] (B5) Obtain constant sequence points

[0105] Repeat step (A) once to obtain a right elbow node transition trajectory sequence, and record the minimum regularized path between the right elbow node transition trajectory sequence generated in the correction operation of step (A3) and the right elbow node rough template trajectory sequence;

[0106] Assume that the number of Euclidean distance values contained in the minimum regularized path is u; each of the u Euclidean distance values is obtained by calculating the Euclidean distance between a sequence point in the transition trajectory sequence of the right elbow node and a sequence point that matches the rough template trajectory sequence of the right elbow node, and the two sequence points used to calculate each Euclidean distance value are recorded as a matching point pair for each Euclidean distance value, obtaining u matching point pairs;

[0107] For each of the u matching point pairs, calculate the weighted average coordinate of each matching point pair based on the two 3D coordinates of each matching point pair according to the weight value e and the weight value f recorded in step (B1), thereby obtaining u weighted average coordinates; set each weighted average coordinate as the 3D coordinate of a temporary sequence point, thereby obtaining u temporary sequence points; the u temporary sequence points correspond to the u Euclidean distance values;

[0108] The u Euclidean distance values are respectively located in the distance matrix of step (A1). The u Euclidean distance values obtained in step (B3) are divided into n groups according to the column number of each Euclidean distance value in the distance matrix. The Euclidean distance values located in the same column of the distance matrix are classified into the same group, and the group number is the same as the column number;

[0109] Divide the u temporary sequence points into n groups, where the u temporary sequence point groups are the same as the u Euclidean distance value groups obtained in step (B3); the distance matrix described in step (A1) has a total of n columns, and the n columns in the distance matrix correspond to the n groups in the u temporary sequence points;

[0110] Calculate the average coordinates of the three-dimensional coordinates of each group of temporary sequence points in n groups of temporary sequence points, set the group number of each group of temporary sequence points to the group value of the calculated average coordinates, and obtain n average coordinates with group values; set each average coordinate of the n average coordinates with group values to the three-dimensional coordinates of a constant point sequence point, and set the group value of each average coordinate to the group value of the corresponding constant point sequence point, and obtain n constant point sequence points with group values;

[0111] (B6) Single constant point template synthesis and replacement

[0112] Sort n constant point sequence points with group values, and place the constant point sequence points with smaller group values before the constant point sequence points with larger group values, to obtain n sorted constant point sequence points; all the sorted n constant point sequence points together constitute a trajectory sequence, which is set as the constant point trajectory sequence;

[0113] Delete the right elbow node rough template trajectory sequence with more than 100 synthesis times, set the constant point trajectory sequence to the right elbow node rough template trajectory sequence with more than 100 synthesis times plus 1, complete one constant point template synthesis and replacement, and obtain the right elbow node rough template trajectory sequence with more than 100 synthesis times plus 1, that is, the right elbow node rough template trajectory sequence with a weight value e of more than 101 plus 1;

[0114] (B7) Obtain the fine template trajectory sequence of the right elbow node

[0115] Repeat steps (B1), (B5), and (B6) to complete 100 constant point template synthesis and replacements, and obtain a rough template trajectory sequence of the right elbow node synthesized 200 times. The weight value e reaches 201. The rough template trajectory sequence of the right elbow node synthesized 200 times is set as the fine template trajectory sequence of the right elbow node, and the fine template trajectory sequence of the right elbow node is obtained.

[0116] The template synthesis and replacement operations of the right wrist node are the same as those of the right elbow node, and a fine template trajectory sequence of the right wrist node is obtained.

[0117] In step (B2) or step (B5), the calculation operation of calculating the weighted average coordinates of each matching point pair based on the two three-dimensional coordinates of each matching point pair is as follows:

[0118] Obtain the first three-dimensional coordinates of the sequence point in the matching point pair from the rough template trajectory sequence of the right elbow node, and multiply the values of the three dimensions of the first three-dimensional coordinates by the weight value e respectively to obtain the first weighted coordinates;

[0119] Obtain the second three-dimensional coordinates of the sequence point in the right elbow node transition trajectory sequence from the matching point pair, and multiply the values of the three dimensions of the second three-dimensional coordinates by the weight value f to obtain a second weighted coordinate;

[0120] The x-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the x-dimensional value in the weighted average coordinate of the matching point pair; the y-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the y-dimensional value in the weighted average coordinate of the matching point pair; the z-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the z-dimensional value in the weighted average coordinate of the matching point pair; the weighted average coordinate of the matching point pair is obtained according to the x-dimensional value, the y-dimensional value and the z-dimensional value.

[0121] In step (B5), the calculation operation of the average coordinates of the three-dimensional coordinates of each group of temporary sequence points is as follows:

[0122] Sum the x-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the x-dimension value of the average coordinates.

[0123] Sum the y-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the y-dimension value of the average coordinates.

[0124] Sum the z-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the z-dimension value of the average coordinates.

[0125] The average coordinates of the three-dimensional coordinates of all temporary sequence points in the same group are obtained according to the x-dimension value, the y-dimension value and the z-dimension value.

[0126] In step (4.1), the improved dynamic time warping algorithm is obtained by adding a correction operation step to the dynamic time warping algorithm. The specific process of the correction operation step is the same as the correction operation step of the improved dynamic time warping algorithm used in the establishment operation of the three-dimensional dynamic gesture classification template library.

[0127] The beneficial technical effects of the present invention are embodied in the following aspects:

[0128] 1. A video recording a three-dimensional dynamic gesture contains a large amount of information, much of which is interference information that is irrelevant to the three-dimensional dynamic gesture to be identified and will interfere with the three-dimensional dynamic gesture recognition. The recognition method of the present invention uses static gestures to issue start and stop instructions to the dynamic gesture recognition system through steps (1) and (2), eliminating redundant actions, leaving only a left video frame and a right video frame containing only the target gesture to be identified, thereby avoiding the interference of redundant actions on dynamic gesture recognition. Through step (3), the two obtained video frames are converted into a three-dimensional motion trajectory sequence of the right elbow node and a three-dimensional motion trajectory sequence of the right wrist node, thereby eliminating the interference information in the complex video frame, leaving only the key information representing the type of dynamic gesture, and reducing the difficulty of completing the three-dimensional dynamic gesture recognition in step (4).

[0129] 2. The three-dimensional dynamic gesture classification template library establishment operation in step (4) of the present invention and the similarity matching operation in step (4.1) both use an improved dynamic time warping algorithm, which adds a correction operation (A3) for the dynamic time warping distance on the basis of the traditional dynamic time warping algorithm steps (A1) and (A2). The improved dynamic time warping algorithm retains the advantage of the traditional dynamic time warping algorithm that it is not easily disturbed by fluctuations in the movement speed of the joint points, while avoiding the interference of the number of sequence points contained in the trajectory sequence on the dynamic time warping distance, making the dynamic time warping distance more accurate, thereby improving the quality of the three-dimensional dynamic gesture classification template library and the accuracy of similarity matching, and further improving the accuracy of three-dimensional dynamic gesture recognition. The present invention pre-designs 6 three-dimensional dynamic gestures, including two simple gestures, two complex gestures, and two more complex combination gestures composed of complex gestures. After establishing a 3D dynamic gesture classification template library for the six 3D dynamic gestures, the tester repeated each 3D dynamic gesture 100 times. Each time, the 3D dynamic gesture recognition system of the present invention performed 3D dynamic gesture recognition, while the 3D dynamic gesture recognition system using the traditional DTW algorithm also performed 3D dynamic gesture recognition, recording the recognition results of both systems. After repeating all 3D dynamic gestures 100 times, the number of times each system correctly recognized the six 3D dynamic gestures was counted. The statistical results were divided by 100 to obtain the recognition accuracy of the two systems for the six 3D dynamic gestures, as shown in Table 1:

[0130] Table 1 Statistics of recognition accuracy of three-dimensional dynamic gestures

[0131]

[0132] In Table 1, system 1 is the three-dimensional dynamic gesture recognition system of the present invention, and system 2 is the three-dimensional dynamic gesture recognition system using the traditional DTW algorithm. As can be seen from the table, the more complex the three-dimensional dynamic gesture to be recognized, the greater the advantage of the present invention in recognition accuracy, with the highest improvement in recognition accuracy of 15%. This is because when the gesture becomes more complex, the number of sequence points contained in the three-dimensional motion trajectory sequence obtained in step (3) will increase significantly, causing interference with the calculation results of the traditional dynamic time warping algorithm. The improved dynamic time warping algorithm of the present invention corrects the interference, allowing the three-dimensional dynamic gesture recognition system to have a higher accuracy rate for complex three-dimensional dynamic gestures.

[0133] 3. Due to the fluctuation of the skeleton recognition model in step (1) during operation, the motion trajectory sequence obtained each time in step (3) will lose some action details, and thus the rough trajectory sequence in step (4) lacks some action details. In order to avoid the influence of the lost action details on the three-dimensional dynamic gesture recognition results, the present invention proposes a three-dimensional dynamic gesture classification template library establishment operation in step (4). The first 100 times of point-increasing template synthesis increases the number of sequence points contained in the rough template trajectory sequence, thereby enriching the trajectory information contained in the rough template trajectory sequence, making the trajectory expressed by the rough template trajectory sequence from rough to fine. Since the system response time will be slow when the number of sequence points contained in the rough template trajectory sequence is too large, the present invention designs the subsequent 100 times of constant point template synthesis to avoid excessive increase in the number of sequence points contained in the rough template trajectory sequence, while maintaining the total number of sequence points unchanged, the three-dimensional coordinates of the sequence points are finely corrected, further enriching the detail information, thereby obtaining the final fine template trajectory sequence. The present invention pre-designs 6 three-dimensional dynamic gestures, including two simple gestures, two complex gestures, and two more complex combination gestures composed of complex gestures. After establishing a 3D dynamic gesture classification template library for the six 3D dynamic gestures, the tester repeated each 3D dynamic gesture 100 times. Each time, the 3D dynamic gesture recognition system of the present invention performed 3D dynamic gesture recognition, while the 3D dynamic gesture recognition system without the 3D dynamic gesture classification template library also performed 3D dynamic gesture recognition, recording the recognition results of both systems. After repeating all 3D dynamic gestures 100 times, the number of times each system correctly recognized the six 3D dynamic gestures was counted, and the statistical results were divided by 100 to obtain the recognition accuracy of the two systems for the six 3D dynamic gestures, as shown in Table 2:

[0134] Table 2 Statistics of recognition accuracy of three-dimensional dynamic gestures

[0135]

[0136] System 1 in Table 2 represents the 3D dynamic gesture recognition system of the present invention, while System 2 represents a 3D dynamic gesture recognition system without a 3D dynamic gesture classification template library. As can be seen from Table 2, the more complex the system, the greater the accuracy advantage of the 3D dynamic gesture recognition system of the present invention, with up to a 10% improvement in recognition accuracy. This is because the more complex the 3D dynamic gesture, the more details are lost in the rough template trajectory sequence. The 3D dynamic gesture classification template library obtained through template synthesis in the present invention supplements these details, thereby improving the recognition accuracy of the 3D dynamic gesture recognition system for complex 3D dynamic gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] Figure 1 is a schematic diagram of the skeleton recognition model in Example 1 of the present invention;

[0138] Figure 2 1 is a diagram showing the image processing process of the left palm image in Example 1 of the present invention;

[0139] Figure 3 is a flowchart of three-dimensional dynamic gesture recognition in Examples 1 and 2 of the present invention;

[0140] Figure 4 This is a flow chart of 200 template syntheses in Example 2 of the present invention;

[0141] Figure 5 This is a schematic diagram of point-added template synthesis in Example 2 of the present invention. The thick lines connecting the sequence points represent that the connected sequence points are from the same trajectory sequence, and the thin lines connecting the sequence points represent the matching relationship between the sequence points.

[0142] Figure 6 Schematic diagram of constant point template synthesis in Example 2 of the present invention. The thick lines connecting the sequence points represent that the connected sequence points are from the same trajectory sequence, and the thin lines connecting the sequence points represent the matching relationship between the sequence points.

[0143] Figure 7 This is a diagram showing the trajectory shape changes of each right elbow template trajectory sequence in the three-dimensional dynamic gesture classification template library in Example 2 of the present invention during the template synthesis process. DETAILED DESCRIPTION

[0144] The present invention will be further described below with reference to the accompanying drawings through embodiments.

[0145] Example 1

[0146] The hardware system of this embodiment 1 includes a left camera, a right camera and a host computer that are parallel to each other, and the spacing between the lens of the left camera and the lens of the right camera is 120 mm. The left camera and the right camera constitute a binocular depth camera, and the field of view of the binocular depth camera is 110° horizontal viewing angle × 70° vertical viewing angle × 120° diagonal viewing angle. The binocular depth camera is connected to the host computer. The host computer is a host computer, and a visual processor (GPU) that supports deep learning is installed in the host computer.

[0147] The specific steps to achieve complete gesture recognition based on the improved DTW algorithm are as follows:

[0148] (1) Left-hand static gesture recognition

[0149] (1.1) Gesture segmentation

[0150] See also Figure 1 , the left camera and the right camera of the binocular depth camera shoot at the same time, and transmit the captured left and right images to the host computer. The host computer imitates the principle of human eyes to judge the distance of objects, and by comparing the differences between the left and right images, calculates the three-dimensional coordinates of each point in the left image relative to the midpoint of the line connecting the left and right cameras, and obtains a three-dimensional point cloud containing three-dimensional coordinate information; based on the three-dimensional point cloud, when the human body appears in the left and right images, a three-dimensional point cloud of the human body is obtained; the visual processor performs deep learning on the three-dimensional point cloud of the human body to realize human skeleton recognition, and obtains the three-dimensional coordinates of the 18 joint nodes of the human body in the left image and the pixel coordinates of the 18 joint nodes in the left image, that is, Figure 1 The skeleton recognition model shown.

[0151] See also Figure 2 , from the skeleton recognition model, select the left wrist node as the basis for locating the left hand position, obtain the three-dimensional coordinates of the left wrist node and the pixel coordinates in the left picture. In the left picture, intercept the left palm image in a rectangular area above the pixel coordinates of the left wrist node, and separate the left palm image from the left picture to obtain Figure 2 The left palm image shown in A in FIG. 1 only contains the palm part of the left hand, completing the gesture segmentation of static gesture recognition.

[0152] (1.2) Image preprocessing

[0153] See also Figure 2, the host computer converts the color gamut of the left palm image from the three primary colors of red, green, and blue (RGB) to hue, saturation, and value (HSV), and regards the pixels with H <= 19, S >= 48, and V > 40 in the hue, saturation, and value as skin pixels, and those that do not meet the skin pixels are regarded as background pixels, so as to binarize the left palm image to distinguish between skin pixels and background pixels; then perform morphological dilation on the binarized left palm image to eliminate the skin pixels misjudged as background in the palm; then perform median filtering to eliminate the jagged contours at the palm edge and smooth the edge; finally, perform contour extraction to eliminate the skin pixels and background pixels outside the palm edge contour, and obtain Figure 2 the complete and clear left palm contour picture shown in B of

[0154] (1.3) Static gesture classification

[0155] The host computer detects multiple structures that are concave inward on the contour of the left palm contour picture. The structure that is concave inward is called a convex hull defect, and records the number of large convex hull defects among all convex hull defects. The large convex hull defect refers to a convex hull defect with a depression depth exceeding one-fourth of the palm length. The number of large convex hull defects plus one is the number of fingers extended in this left palm contour, and the static gesture is classified according to the number of fingers extended.

[0156] See Figure 2 , select Figure 2 the static gesture "one" with one finger extended shown in C of Figure 2 as the stop instruction for dynamic gesture recognition, and

[0157] (2) Eliminate redundant actions

[0158] Before starting the three-dimensional dynamic gesture recognition, first start the static gesture recognition of the left hand, and regard the static gesture "five" with five fingers extended as the start instruction for the three-dimensional dynamic gesture recognition. After the tester makes the gesture "five" with the left hand, the three-dimensional dynamic gesture recognition officially starts.

[0159] Regard the static gesture "one" with one finger extended as the stop instruction for the three-dimensional dynamic gesture recognition, and stop the three-dimensional dynamic gesture recognition after the tester makes the gesture "one" with the left hand.

[0160] Regard the dynamic actions between these two static gestures as the three-dimensional dynamic gestures that need to be recognized, and ignore the redundant dynamic gestures that are not desired to be recognized.

[0161] (3) Obtain the three-dimensional motion trajectory sequences of the right wrist node and the right elbow node

[0162] When receiving the 3D dynamic gesture recognition start command, the host computer continuously receives the left and right video images sent back by the binocular depth camera, and continuously identifies and tracks the left shoulder node, right shoulder node, right elbow node, and right wrist node in the left video image through the same skeleton recognition model as step (1.1), and obtains four 3D original trajectory sequences of the left shoulder node, right shoulder node, right elbow node, and right wrist node. The four 3D original trajectory sequences are preprocessed to obtain the 3D motion trajectory sequences of the right wrist node and right elbow node. The preprocessing steps are as follows:

[0163] (3.1) Obtaining the original three-dimensional trajectory sequence of joint points

[0164] The left and right cameras of the binocular depth camera continuously shoot simultaneously and transmit the left and right video images back to the host computer. The host computer uses the same skeleton recognition model as step (1.1) to identify and track the four nodes in the left video image: the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node. A total of 200 consecutive frames of images are selected as subsequent processing images.

[0165] Obtain four three-dimensional coordinates in each of the 200 frames of images, which are the three-dimensional coordinates of the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node relative to the midpoint of the line connecting the left camera and the right camera. One three-dimensional coordinate represents a sequence point, and 200 sequence points of each node in the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node are obtained respectively; the 200 sequence points of the left shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain the three-dimensional original trajectory sequence of the left shoulder node; the 200 sequence points of the right shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain the three-dimensional original trajectory sequence of the right shoulder node; the 200 sequence points of the right elbow shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain the three-dimensional original trajectory sequence of the right elbow node; the 200 sequence points of the right wrist node are arranged from front to back according to the time sequence in the 200 frames of images to obtain the three-dimensional original trajectory sequence of the right wrist node, and a total of four three-dimensional original trajectory sequences are obtained.

[0166] (3.2) Obtain the three-dimensional relative trajectory sequence of the right wrist node and the right elbow node

[0167] Set a variable q with an initial value of 1. Take the horizontal projection of the line connecting the qth sequence point in the three-dimensional original trajectory sequence of the left shoulder node and the qth sequence point in the three-dimensional original trajectory sequence of the right shoulder node as the x-axis, set the vertical direction as the z-axis, and construct a right-handed Cartesian coordinate system as the qth human coordinate system according to the right-hand rule of the Cartesian coordinate system; calculate the relative coordinates of the qth sequence point in the three-dimensional original trajectory sequence of the right wrist node in the qth human coordinate system through the coordinate transformation formula, which represents the qth sequence point in the three-dimensional relative trajectory sequence of the right wrist node.

[0168] The coordinate transformation formula is as follows:

[0169]

[0170]

[0171]

[0172]

[0173] In formula (1)(2)(3)(4), (x A ,y A ,z A ) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the left shoulder node, (x B ,y B ,z B ) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the right shoulder node, (x, y, z) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the right elbow node or the qth sequence point in the three-dimensional original trajectory sequence of the right wrist node that requires coordinate transformation, (x′, y′, z′) is the three-dimensional relative coordinate of the qth sequence point in the three-dimensional relative trajectory sequence of the right elbow node or the qth sequence point in the three-dimensional relative trajectory sequence of the right wrist node that requires relative coordinate calculation in the qth human body relative coordinate system, and the unit of all coordinates is m.

[0174] The relative coordinates of the qth sequence point in the qth human body coordinate system in the three-dimensional original trajectory sequence of the right elbow node are calculated using the same coordinate transformation formula as that of the right wrist, representing the qth sequence point in the three-dimensional relative trajectory sequence of the right elbow node.

[0175] Keep increasing the q value by one and repeat step (3.2) until the q value reaches 200, and obtain the three-dimensional relative trajectory sequence of the right wrist node and the three-dimensional relative trajectory sequence of the right elbow node.

[0176] (3.3) Obtain the three-dimensional motion trajectory sequence of the right wrist node and the right elbow node respectively

[0177] The wrist distances between adjacent sequence points in the three-dimensional relative trajectory sequence of the right wrist node are calculated, and the elbow distances between adjacent sequence points in the three-dimensional relative trajectory sequence of the right elbow node are calculated.

[0178] The movement speed of each sequence point in the three-dimensional relative trajectory sequence of the right wrist node is calculated based on the wrist distance. Based on the movement speed, each sequence point in the three-dimensional relative trajectory sequence of the right wrist node is classified into high-speed points and low-speed points using a clustering algorithm. The specific steps are as follows:

[0179] (3.3.1) Calculate the velocity of all sequence points in the three-dimensional relative trajectory sequence based on the wrist distance. Randomly select two sequence points a and b in the three-dimensional relative trajectory sequence and calculate the difference between the velocity of all sequence points in the entire sequence trajectory and the velocity of sequence points a and b. If the velocity difference between a sequence point and sequence point a is smaller than the velocity difference between a sequence point and sequence point b, then classify the sequence point as Class A. If the velocity difference between a sequence point and sequence point a is larger than the velocity difference between a sequence point and sequence point b, then classify the sequence point as Class B.

[0180] (3.3.2) Calculate the average speed of A for category A and the average speed of B for category B. Find the sequence point in category A whose speed is closest to the average speed of A and use this sequence point to replace the original sequence point a. Find the sequence point in category B whose speed is closest to the average speed of B and use this sequence point to replace the original sequence point b. Repeat step (3.3.1) to reclassify all sequence points.

[0181] (3.3.3) Repeat step (3.3.2) at least twice, and check whether sequence point a and sequence point b are replaced each time. If sequence point a and sequence point b are no longer replaced with the repetition of step (3.3.2), it means that the optimal classification has been achieved at this time. Stop repeating step (3.3.2) and regard the class with the larger average speed between class A and class B as the high-speed class, and all sequence points in this class are called high-speed points. The class with the smaller average speed is regarded as the low-speed class, and all sequence points in this class are called low-speed points.

[0182] The movement speed of each sequence point in the three-dimensional relative trajectory sequence of the right elbow node is calculated based on the elbow distance. Based on the movement speed, the sequence points in the three-dimensional relative trajectory sequence of the right elbow node are classified into high-speed points and low-speed points using a clustering algorithm. The specific steps are the same as those for classifying the sequence points in the three-dimensional relative trajectory sequence of the right wrist.

[0183] The continuously distributed low-speed points are set as the static stage, and the continuously distributed high-speed points are set as the moving stage. The static stage is eliminated, and the three-dimensional motion trajectory sequence of the right wrist node containing only the moving stage and the three-dimensional motion trajectory sequence of the right elbow node containing only the moving stage are obtained respectively.

[0184] The continuous distribution means that: more than five consecutive points in the three-dimensional relative trajectory sequence of the right elbow node are simultaneously classified as low-speed points or simultaneously classified as high-speed points; or more than five consecutive points in the three-dimensional relative trajectory sequence of the right wrist node are simultaneously classified as low-speed points or simultaneously classified as high-speed points.

[0185] (4) 3D dynamic gesture recognition

[0186] See also Figure 3 Based on a 3D dynamic gesture classification template library containing four 3D dynamic gestures, namely complex gesture 1, complex gesture 2, combined gesture 1, and combined gesture 2, the specific steps of dynamic gesture recognition are as follows:

[0187] (4.1) Obtaining the most similar dynamic gesture

[0188] The tester imitates complex gesture 1 and makes a 3D dynamic gesture to be recognized. According to steps (2) and (3), the motion trajectory sequences of the right wrist node and the right elbow node are obtained. Then, the motion trajectory sequences of the two joint points are matched with each 3D dynamic gesture classification template in the 3D dynamic gesture classification template library by using the improved dynamic time warping algorithm. The specific steps of similarity matching are as follows:

[0189] The improved dynamic time warping distance between the motion trajectory sequence of the right wrist node to be identified and the fine template trajectory sequence of the right wrist node in the three-dimensional dynamic gesture classification template is calculated to obtain the improved dynamic time warping distance of the right wrist node.

[0190] The improved dynamic time warping distance between the motion trajectory sequence of the right elbow node to be identified and the fine template trajectory sequence of the right elbow node in the three-dimensional dynamic gesture classification template is calculated to obtain the improved dynamic time warping distance of the right elbow node.

[0191] The average of the two improved dynamic time warping distances is used as the similarity evaluation value between the motion trajectory sequences of the two joints to be identified and the 3D dynamic gesture classification template. The smaller the similarity evaluation value, the greater the similarity. The similarity evaluation values obtained from the 3D dynamic gesture classification template for complex gesture 1, complex gesture 2, combined gesture 1, and combined gesture 2 were 0.102m, 0.436m, 0.607m, and 0.554m, respectively.

[0192] The 3D dynamic gesture classification template that is most similar to the motion trajectory sequence to be identified is screened out from all the 3D dynamic gesture classification templates in the 3D dynamic gesture classification template library. The most similar 3D dynamic gesture classification template is the 3D dynamic gesture classification template of complex gesture 1. Complex gesture 1 is recorded as the most similar dynamic gesture, and the corresponding similarity evaluation value of 0.102m is called the minimum dynamic time warping distance.

[0193] The improved dynamic time warping algorithm is obtained by adding a correction operation step to the dynamic time warping algorithm.

[0194] (4.2) Obtaining the results of three-dimensional dynamic gesture recognition

[0195] The minimum dynamic time warping distance value is 0.102, which is less than 0.2m. The 3D dynamic gesture is successfully recognized. At this time, the most similar dynamic gesture, namely complex gesture 1, is the gesture recognition result. The recognition result is consistent with the gesture made by the tester, that is, the 3D dynamic gesture recognition result is correct.

[0196] After this 3D dynamic gesture recognition is completed, the system starts waiting for the next 3D dynamic gesture recognition.

[0197] Example 2: Establishing a three-dimensional dynamic gesture classification template library

[0198] See also Figure 4 This embodiment 2 is the operation of establishing the three-dimensional gesture classification template library used in step (4) of embodiment 1. The hardware system used is the same as that of embodiment 1. The specific operation steps are as follows:

[0199] See also Figure 7 , design a complex gesture 1 as the target three-dimensional dynamic gesture, the tester repeats the target three-dimensional dynamic gesture once, extract the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node through steps (2) and (3) of Example 1, set the three-dimensional motion trajectory of the right wrist node as the rough template trajectory sequence of the right wrist node, and set the three-dimensional motion trajectory sequence of the right elbow node as the rough template trajectory sequence of the right elbow node, and officially start the template synthesis of the target three-dimensional dynamic gesture. The trajectory shape of the rough template trajectory sequence of the right elbow node of the complex gesture 1 before the template synthesis starts is as follows Figure 7 As shown in A1.

[0200] (A) Obtaining transition trajectory sequence

[0201] The tester repeats the target three-dimensional dynamic gesture once, and extracts the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node through steps (2) and (3) of Example 1. The two three-dimensional motion trajectory sequences are respectively matched with the rough template trajectory sequences of the two joint points through the improved dynamic time warping algorithm for similarity.

[0202] The specific operations of right elbow node similarity matching are as follows:

[0203] (A1) Construct an m×n matrix, called a distance matrix, where the element in the i-th row and j-th column of the matrix is the Euclidean distance between the i-th sequence point in the three-dimensional motion trajectory sequence of the right elbow node and the j-th sequence point in the rough template trajectory sequence of the right elbow node.

[0204] The m in the matrix is the number of sequence points contained in the three-dimensional motion trajectory sequence of the right elbow node.

[0205] The n in the matrix is the number of sequence points contained in the rough template trajectory sequence of the right elbow node.

[0206] (A2) Then construct an m×n matrix, called the cumulative distance matrix. The element in the i-th row and j-th column of the cumulative distance matrix is obtained by the recursive formula (5):

[0207] D ij =d ij +min{D i,j-1 ,D i-1,j-1 ,D i-1,j}——(5)

[0208] In formula (5), d ij is the element in row i and column j of the distance matrix, D ij To accumulate the elements in row i and column j of the distance matrix, when row number i or column number j is less than 1, 0 is used instead of D ij , D 11 =d 11 , starting from the element in the first row and first column, calculate all the elements in the cumulative distance matrix from the first column to the last column by recursive formula (5), then the element D in the mth row and nth column is mn The dynamic time warping distance between two trajectory sequences calculated by the traditional dynamic time warping algorithm will increase with the increase of m and n, so it needs to be corrected.

[0209] (A3) The correction steps are as follows:

[0210] In order to correct the influence of the size of m and n on the traditional dynamic time warping distance, from D mn First, recursively follow the recursive formula (5) to find and record all the distance matrices in step (A1) that participated in D mn Calculate the element d ij , these elements d ij The set of is called the minimum warped path between the two trajectory sequences; the elements in each column of the distance matrix that are in the minimum warped path are averaged, the average values of each column are accumulated and divided by the total number of columns n to obtain the improved dynamic time warping distance between the three-dimensional motion trajectory sequence of the right elbow node and the rough template trajectory sequence of the right elbow node; the smaller the improved dynamic time warping distance, the higher the similarity.

[0211] The size of the improved dynamic time warping distance is not affected by the sizes of m and n like the traditional dynamic time warping distance.

[0212] The similarity matching operation of the right wrist node is the same as that of the right elbow node.

[0213] The improved dynamic time warping distance between the three-dimensional motion trajectory sequence of the right wrist node and the rough template trajectory sequence of the right wrist node, and the improved dynamic time warping distance between the three-dimensional motion trajectory sequence of the right elbow node and the rough template trajectory sequence of the right elbow node are obtained, thus completing the similarity calculation.

[0214] The two improved dynamic time warping distances respectively represent the similarity between the three-dimensional motion trajectory sequences of the two joint points and the rough template trajectory sequence. If the two improved dynamic time warping distances obtained are both less than 0.2m, it is considered that the three-dimensional motion trajectory sequences of the two joint points are sufficiently similar to the recorded rough template trajectory sequence. The three-dimensional motion trajectory sequences of the two joint points are called the transition trajectory sequences of the two joint points, that is, the transition trajectory sequence of the right wrist node and the transition trajectory sequence of the right elbow node. If one of the two improved dynamic time warping distances obtained is greater than 0.2m or both are greater than 0.2m, repeat the similarity calculation until the two improved dynamic time warping distances obtained are both less than 0.2m. The three-dimensional motion trajectory sequences of the two joint points are called the transition trajectory sequences of the two joint points, that is, the transition trajectory sequence of the right wrist node and the transition trajectory sequence of the right elbow node.

[0215] (B) Obtaining a fine template trajectory sequence

[0216] The transition trajectory sequence of the right wrist node and the rough template trajectory sequence of the right wrist node are synthesized by the template synthesis algorithm to obtain a new rough template trajectory sequence of the right wrist. The transition trajectory sequence of the right elbow node and the rough template trajectory sequence of the right elbow node are synthesized to obtain a new rough template trajectory sequence of the right elbow. The previous rough template trajectory sequences are replaced and synthesized again. After accumulating synthesis and replacement for 200 times, the fine template trajectory sequences of the right wrist node and the fine template trajectory sequences of the right elbow node are obtained respectively. The specific steps are as follows:

[0217] The template synthesis and replacement operations of the right elbow node are as follows:

[0218] (B1) Obtain two weight values

[0219] The template is synthesized by the rough template trajectory sequence of the right elbow node and the transition trajectory sequence of the right elbow node. The weight value of the rough template trajectory sequence of the right elbow node is set to e, and the weight value of the transition trajectory sequence of the right elbow node is set to f.

[0220] The 3D motion trajectory sequence of the right elbow node recorded when the tester makes the target 3D dynamic gesture for the first time is used as the initial rough template trajectory sequence of the right elbow node, and the weight value e is recorded as 1. Thereafter, the weight value e of the rough template trajectory sequence of the right elbow node is increased by 1 each time the template synthesis is performed.

[0221] The weight value f of the right elbow node transition trajectory sequence in template synthesis is always 1.

[0222] (B2) Obtaining the incremental sequence points

[0223] Repeat step (A) once to obtain a right elbow node transition trajectory sequence, and record the minimum regular path between the right elbow node transition trajectory sequence generated in the correction operation of step (A3) and the right elbow node rough template trajectory sequence.

[0224] See also Figure 5 , let the number of Euclidean distance values contained in the minimum regular path be u; each of the u Euclidean distance values is obtained by calculating the Euclidean distance between a sequence point in the right elbow node transition trajectory sequence and a sequence point matching the right elbow node rough template trajectory sequence, for example Figure 5 The A1 sequence point is a sequence point in the right elbow node transition trajectory sequence, and the B1 sequence point is a sequence point in the right elbow node rough template trajectory sequence that matches the A1 sequence point. The thin line connecting the two sequence points A1B1 represents that this pair of sequence points matches each other. This pair of sequence points calculates one of the Euclidean distance values of u Euclidean matrix values. Figure 5 The other thin lines in the same meaning as the thin lines of A1B1. The two sequence points used to calculate each Euclidean distance value are recorded as the matching point pair of each Euclidean distance value, for example Figure 5 A1B1, A2B2, A2B3, A3B4, A3B5, and A4B6 are each recorded as a matching point pair. u Euclidean distance values correspond to u matching point pairs.

[0225] See also Figure 5 , u Euclidean distance values are respectively located in the distance matrix of step (A1), record the row number and column number of each Euclidean distance value in the distance matrix, set the row number of each Euclidean distance value in the distance matrix to the row value of the matching point pair of each Euclidean distance value, and set the column number of each Euclidean distance value in the distance matrix to the column value of the matching point pair of each Euclidean distance value, for example Figure 5 The Euclidean distance value calculated for the matching point pair A2B3 is located in row number 3 and column number 2 in the distance matrix, so the row value of the matching point pair A2B3 is 3 and the column value is 2.

[0226] For each of the u matching point pairs, calculate the weighted average coordinates of each matching point pair based on the two 3D coordinates of each matching point pair according to the weight values e and f recorded in step (B1), and obtain u weighted average coordinates. The calculation operation of the weighted average coordinates of the matching point pairs is as follows:

[0227] Obtain the first three-dimensional coordinates of the sequence point in the matching point pair from the rough template trajectory sequence of the right elbow node, and multiply the values of the three dimensions of the first three-dimensional coordinates by the weight value e respectively to obtain the first weighted coordinates;

[0228] Obtain the second three-dimensional coordinates of the sequence point in the right elbow node transition trajectory sequence from the matching point pair, and multiply the values of the three dimensions of the second three-dimensional coordinates by the weight value f to obtain a second weighted coordinate;

[0229] The x-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the x-dimensional value in the weighted average coordinate of the matching point pair; the y-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the y-dimensional value in the weighted average coordinate of the matching point pair; the z-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the z-dimensional value in the weighted average coordinate of the matching point pair; the weighted average coordinate of the matching point pair is obtained according to the x-dimensional value, the y-dimensional value and the z-dimensional value.

[0230] See also Figure 5 , set each weighted average coordinate to the three-dimensional coordinate of an incremental sequence point, for example Figure 5 In the example, the weighted average coordinates of the matching point pair A1B1 are set as the three-dimensional coordinates of the incremental sequence point C1 to obtain the incremental sequence point C1. Figure 5 The other incremental sequence points C2, C3, C4, C5, and C6 are obtained in the same manner as the incremental sequence point C1. There are a total of u matching point pairs, and u incremental sequence points are obtained. According to the correspondence between the u incremental sequence points and the u matching points, the row value of each matching point pair is set to the row value of the corresponding incremental sequence point, and the column value of each matching point pair is set to the column value of the corresponding incremental sequence point, thus obtaining u incremental sequence points with row and column values.

[0231] (B3) Single-point template synthesis and replacement

[0232] Sort u increment sequence points with row values and column values, arranging increment sequence points with smaller row values before increment sequence points with larger row values, and arranging increment sequence points with smaller column values before increment sequence points with larger column values, to obtain sorted u increment sequence points; all sorted u increment sequence points together constitute a trajectory sequence, which is set as the increment trajectory sequence;

[0233] Delete the right elbow node rough template trajectory sequence, set the added point trajectory sequence to the right elbow node rough template trajectory sequence with the number of synthesis increased by one, complete one point-added template synthesis and replacement, and obtain the right elbow node rough template trajectory sequence with the number of times participating in the synthesis increased by one, that is, the right elbow node rough template trajectory sequence with the weight value e increased by one.

[0234] (B4) Complete 100 times of template synthesis

[0235] See also Figure 7 Repeat steps (B1), (B2) and (B3) to complete one hundred times of template synthesis and replacement, and obtain the rough template trajectory sequence of the right elbow node with one hundred synthesis times. The weight value e reaches 101. The trajectory shape of the rough template trajectory sequence of the right elbow node with one hundred synthesis times is as follows: Figure 7 As shown in A2.

[0236] (B5) Obtain constant sequence points

[0237] Repeat step (A) once to obtain a right elbow node transition trajectory sequence, and record the minimum regularized path between the right elbow node transition trajectory sequence generated in the correction operation of step (A3) and the right elbow node rough template trajectory sequence;

[0238] See also Figure 6 , let the number of Euclidean distance values contained in the minimum regular path be u; each of the u Euclidean distance values is obtained by calculating the Euclidean distance between a sequence point in the right elbow node transition trajectory sequence and a sequence point matching the right elbow node rough template trajectory sequence, for example Figure 6 The A1 sequence point is a sequence point in the right elbow node transition trajectory sequence, and the B1 sequence point is a sequence point in the right elbow node rough template trajectory sequence that matches the A1 sequence point. The thin line connecting the two sequence points A1B1 represents that this pair of sequence points matches each other. This pair of sequence points calculates one of the Euclidean distance values of u Euclidean matrix values. Figure 6 The other thin lines in the same meaning as the thin lines of A1B1. The two sequence points used to calculate each Euclidean distance value are recorded as the matching point pair of each Euclidean distance value, for example Figure 6 A1B1, A2B2, A2B3, A3B4, A3B5, and A4B6 are each recorded as a matching point pair. u Euclidean distance values correspond to u matching point pairs.

[0239] For each of the u matching point pairs, according to the weight value e and weight value f recorded in step (B1), the weighted average coordinate of each matching point pair is calculated based on the two three-dimensional coordinates of each matching point pair to obtain u weighted average coordinates; the specific calculation process of the weighted average coordinate is the same as step (2).

[0240] See also Figure 6 , set each weighted average coordinate as the three-dimensional coordinate of a temporary sequence point, and obtain u temporary sequence points, for example Figure 6 In the example, the weighted average coordinates of the matching point pair A1B1 are set as the three-dimensional coordinates of the temporary sequence point C1 to obtain the temporary sequence point C1. Figure 6 The other temporary sequence points C2, C3, C4, C5 and C6 are obtained in the same way as the temporary sequence point C1. u Euclidean distance values can obtain u temporary sequence points, and u temporary sequence points correspond to u Euclidean distance values.

[0241] See also Figure 6 , u Euclidean distance values are respectively located in the distance matrix of step (A1). According to the column number of each Euclidean distance value in the distance matrix, the u Euclidean distance values obtained in step (B3) are divided into n groups. The Euclidean distance values located in the same column of the distance matrix are classified into the same group, and the group number is the same as the column number, for example Figure 6 In the distance matrix, the Euclidean distance values corresponding to the matching point pair A2B2 and the Euclidean distance values corresponding to the matching point pair A2B3 are both 2 in column number, and are classified into the same group, group number 2; the Euclidean distance values corresponding to the matching point pair A3B4 and the Euclidean distance values corresponding to the matching point pair A3B5 are both 3 in column number, and are classified into the same group, group number 3;

[0242] See also Figure 6 , divide u temporary sequence points into n groups, and the u temporary sequence point groups are the same as the u Euclidean distance value groups obtained in step (B3). Figure 6 In the example, temporary sequence points C2 and C3 are grouped into the same group, group number 2; temporary sequence points C4 and C5 are grouped into the same group, group number 3. The distance matrix described in step (A1) has n columns, and the n columns in the distance matrix correspond to the n groups in the u temporary sequence points.

[0243] See also Figure 6 , calculate the average coordinates of the three-dimensional coordinates of each group of temporary sequence points in n groups of temporary sequence points, set the group number of each group of temporary sequence points to the group value of the calculated average coordinates, and obtain n average coordinates with group values; set each average coordinate of the n average coordinates with group values to the three-dimensional coordinates of a constant point sequence point, and set the group value of each average coordinate to the group value of the corresponding constant point sequence point, and obtain n constant point sequence points with group values. For example Figure 6 In the above, the average coordinates of the second group of temporary sequence points are set as the three-dimensional coordinates of the constant sequence point D1, and the group value 2 is set as the group value of D1; the average coordinates of the third group of temporary sequence points are set as the three-dimensional coordinates of the constant sequence point D2, and the group value 3 is set as the group value of D2.

[0244] The calculation operation of the weighted average coordinates of the three-dimensional coordinates of each group of new sequence points is as follows:

[0245] Sum the x-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the x-dimension value of the average coordinates.

[0246] Sum the y-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the y-dimension value of the average coordinates.

[0247] Sum the z-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the z-dimension value of the average coordinates.

[0248] The average coordinates of the three-dimensional coordinates of all temporary sequence points in the same group are obtained according to the x-dimension value, the y-dimension value and the z-dimension value.

[0249] (B6) Single constant point template synthesis and replacement

[0250] Sort n constant point sequence points with group values, and place the constant point sequence points with smaller group values before the constant point sequence points with larger group values, to obtain n sorted constant point sequence points; all the sorted n constant point sequence points together constitute a trajectory sequence, which is set as the constant point trajectory sequence;

[0251] Delete the right elbow node rough template trajectory sequence with more than 100 synthesis times, set the constant point trajectory sequence to the right elbow node rough template trajectory sequence with more than 100 synthesis times plus 1, complete one constant point template synthesis and replacement, and obtain the right elbow node rough template trajectory sequence with more than 100 synthesis times plus 1, that is, the right elbow node rough template trajectory sequence with a weight value e of more than 101 plus 1;

[0252] (B7) Obtain the fine template trajectory sequence of the right elbow node

[0253] See also Figure 7 Repeat steps (B1), (B5) and (B6) to complete 100 constant point template synthesis and replacement, and obtain the rough template trajectory sequence of the right elbow node with a synthesis time of 200 times. The weight value e reaches 201. The rough template trajectory sequence of the right elbow node with a synthesis time of 200 times is set as the fine template trajectory sequence of the right elbow node to obtain the fine template trajectory sequence of the right elbow node. The trajectory shape of the fine template trajectory sequence of the right elbow node is as follows: Figure 7 As shown in (A3) in .

[0254] The template synthesis and replacement operations for the right wrist node are the same as those for the right elbow node, obtaining a fine template trajectory sequence for the right wrist node. The fine template trajectory sequence for the right wrist node and the fine template trajectory sequence for the right elbow node are recorded as a set of fine template trajectory sequences.

[0255] (C) Obtaining a 3D dynamic gesture classification template library

[0256] The set of fine template trajectory sequences obtained in step (B) is called a first 3D dynamic gesture classification template. The host computer stores the first 3D dynamic gesture classification template in a blank file, which is called a 3D dynamic gesture classification template library.

[0257] See also Figure 7 , design a complex gesture 2 as the second target 3D dynamic gesture, repeat steps (A) and (B) to obtain a set of fine template trajectory sequences of the second target 3D dynamic gesture, set it as the second 3D dynamic gesture classification template, and store it in the 3D dynamic gesture classification template library. The trajectory shape of the rough template trajectory sequence of the right elbow node of complex gesture 2 before template synthesis is as follows Figure 7 As shown in (B1), the trajectory shape of the rough template trajectory sequence of the right elbow node of complex gesture 2 with synthesis times 100 is as follows Figure 7 As shown in (B2), the trajectory shape of the fine template trajectory sequence of the right elbow node of complex gesture 2 is as follows Figure 7 As shown in (B3).

[0258] See also Figure 7 The 3D dynamic gesture formed by performing complex gesture 1 and then performing complex gesture 2 immediately is set as combined gesture 1, which is the third target 3D dynamic gesture. Steps (A) and (B) are repeated to obtain a set of fine template trajectory sequences for the third target 3D dynamic gesture. These are set as the third 3D dynamic gesture classification template and stored in the 3D dynamic gesture classification template library. The trajectory shape of the rough template trajectory sequence of the right elbow node of complex gesture 2 before template synthesis is as follows: Figure 7 As shown in (C1), the trajectory shape of the rough template trajectory sequence of the right elbow node of complex gesture 2 with a synthesis of 100 times is as follows Figure 7 As shown in (C2), the trajectory shape of the fine template trajectory sequence of the right elbow node of complex gesture 2 is as follows Figure 7 As shown in (C3).

[0259] See also Figure 7, the 3D dynamic gesture composed of repeating complex gesture 2 twice is set as combined gesture 2, as the fourth target 3D dynamic gesture, repeat steps (A) and (B) to obtain a set of fine template trajectory sequences of the fourth target 3D dynamic gesture, set as the fourth 3D dynamic gesture classification template, and store it in the 3D dynamic gesture classification template library. The trajectory shape of the rough template trajectory sequence of the right elbow node of complex gesture 2 before template synthesis is as follows Figure 7 As shown in (D1), the trajectory shape of the rough template trajectory sequence of the right elbow node of complex gesture 2 with a synthesis time of 100 is as follows Figure 7 As shown in (D2), the trajectory shape of the fine template trajectory sequence of the right elbow node of complex gesture 2 is as follows Figure 7 As shown in (D3).

[0260] At this time, the number of three-dimensional dynamic gestures recorded in the three-dimensional dynamic gesture classification template library reaches four, and a three-dimensional dynamic gesture classification template library that can be used as a basis for three-dimensional dynamic gesture classification is obtained.

[0261] Example 3

[0262] This embodiment 3 is a three-dimensional dynamic gesture recognition process. The hardware system used in this embodiment 3 is the same as that in embodiment 1. Steps (1), (2), and (3) of this embodiment 3 are completely consistent with steps (1), (2), and (3) of embodiment 1.

[0263] (4) 3D dynamic gesture recognition

[0264] See also Figure 3 According to the same three-dimensional dynamic gesture classification template library as in Example 1, which includes four three-dimensional dynamic gestures, namely complex gesture 1, complex gesture 2, combined gesture 1, and combined gesture 2, the specific operation steps of dynamic gesture recognition are as follows:

[0265] (4.1) Obtaining the most similar dynamic gesture

[0266] The tester imitates the combined gesture 1 to make a three-dimensional dynamic gesture to be recognized, obtains the motion trajectory sequences of the right wrist node and the right elbow node according to steps (2) and (3), and performs a similarity match between the motion trajectory sequences of the two joint points and each three-dimensional dynamic gesture classification template in the three-dimensional dynamic gesture classification template library through the improved dynamic time warping algorithm. The specific operation of the similarity matching is the same as step (4.1) of Example 1. Finally, the similarity evaluation values of the three-dimensional dynamic gesture classification templates of complex gesture 1, complex gesture 2, combined gesture 1, and combined gesture 2 are 0.650m, 0.755m, 0.095m, and 0.443m, respectively.

[0267] The 3D dynamic gesture classification template that is most similar to the motion trajectory sequence to be identified is screened out from all the 3D dynamic gesture classification templates in the 3D dynamic gesture classification template library. The most similar 3D dynamic gesture classification template is the 3D dynamic gesture classification template of combination gesture 1. Combination gesture 1 is recorded as the most similar dynamic gesture, and the corresponding similarity evaluation value of 0.095m is called the minimum dynamic time warping distance.

[0268] The improved dynamic time warping algorithm is the same as that in Example 1.

[0269] (4.2) Obtaining the results of three-dimensional dynamic gesture recognition

[0270] The minimum dynamic time warping distance value is 0.095m, which is less than 0.2m. The 3D dynamic gesture is successfully recognized. At this time, the most similar dynamic gesture, namely combined gesture 1, is the gesture recognition result. The recognition result is consistent with the gesture made by the tester, that is, the 3D dynamic gesture recognition result is correct.

[0271] It will be easily understood by those skilled in the art that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for three-dimensional dynamic gesture recognition based on an improved dynamic time warping algorithm. The hardware system implementing this method includes a left camera, a right camera, and a host computer. The left and right cameras form a binocular depth camera, which is connected to the host computer. The host computer is also known as a host computer. The host computer is equipped with a deep learning-supported visual processing unit (GPU). Its characteristics are: The specific steps to achieve complete gesture recognition based on the improved dynamic time warping algorithm are as follows: (1) Static gesture recognition of left-hand gestures (1.1) Gesture segmentation The left and right cameras of the binocular depth camera simultaneously shoot and transmit the captured left and right images to a host computer. The host computer, based on the principle of imitating the human eye's judgment of object distance, compares the differences between the left and right images and calculates the three-dimensional coordinates of each point in the left image relative to the midpoint of the line connecting the left and right cameras, thereby obtaining a three-dimensional point cloud containing three-dimensional coordinate information. Based on the three-dimensional point cloud, when a human body appears in the left and right images, a three-dimensional point cloud of the human body is obtained. The visual processor performs deep learning on the three-dimensional point cloud of the human body to realize human skeleton recognition, obtains the three-dimensional coordinates of the 18 joint nodes of the human body in the left image and the pixel coordinates of the 18 joint nodes in the left image, and thus obtains a skeleton recognition model. From the skeleton recognition model, the left wrist node is selected as the basis for locating the left hand position, and the three-dimensional coordinates of the left wrist node and the pixel coordinates in the left image are obtained; the left palm image within a rectangular area above the pixel coordinates of the left wrist node in the left image is intercepted, and the left palm image is segmented from the left image to obtain a left palm image containing only the left palm portion, completing the gesture segmentation of static gesture recognition; (1.2) Image preprocessing The host computer converts the color gamut of the left palm image from the three primary colors of red, green and blue (RGB) to the hue-saturation value (HSV), and considers pixels with hue-saturation values of H<=19, S>=48, and V>40 as skin pixels, and pixels that do not meet the hue-saturation value as background pixels, thereby binarizing the left palm image to distinguish between skin pixels and background pixels. Then, morphological dilation is performed on the binarized left palm image to remove skin pixels that are mistakenly identified as background. Median filtering is then performed to remove the jagged edges of the palm and smooth the edges. Finally, contour extraction is performed to remove skin pixels and background pixels outside the palm edge contour to obtain a complete and clear left palm contour image, completing image preprocessing. (1.3) Static gesture classification The host computer detects multiple inward-facing concave structures on the left palm outline image. These inward-facing concave structures are called convex hull defects. The number of large convex hull defects among all convex hull defects is recorded. The number of large convex hull defects plus one is the number of extended fingers in the left palm outline. Static gestures are classified according to the number of extended fingers. The static gesture "one" of extending one finger is selected as the stop instruction for dynamic gesture recognition, and the static gesture "five" of extending five fingers is selected as the start instruction for dynamic gesture recognition; (2) Eliminate redundant actions Before starting 3D dynamic gesture recognition, first start static gesture recognition for the left hand. Use the static gesture "five" of extending five fingers as the starting instruction for 3D dynamic gesture recognition. After the tester makes the static gesture "five" with his left hand, 3D dynamic gesture recognition officially begins. The static gesture "1" of extending one finger is used as the stop instruction for 3D dynamic gesture recognition. The 3D dynamic gesture recognition is stopped after the tester makes the static gesture "1" with his left hand. The dynamic action between the two static gestures is regarded as a 3D dynamic gesture that needs to be recognized, and the redundant dynamic gestures that are not desired to be recognized are ignored; (3) Obtain the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node When receiving the 3D dynamic gesture recognition start command, the host computer continuously receives the left and right video images sent back by the binocular depth camera, and continuously identifies and tracks the left shoulder node, right shoulder node, right elbow node, and right wrist node in the left video image through the skeleton recognition model in step (1.1), and obtains four 3D original trajectory sequences of the left shoulder node, right shoulder node, right elbow node, and right wrist node. The four 3D original trajectory sequences are preprocessed to obtain the 3D motion trajectory sequence of the right wrist node and the 3D motion trajectory sequence of the right elbow node, respectively. The preprocessing operation steps are as follows: (3.1) Obtaining the original three-dimensional trajectory sequence of joint points The left and right cameras of the binocular depth camera continuously shoot simultaneously and transmit the left and right video images back to the host computer. The host computer uses the skeleton recognition model in step (1.1) to identify and track the four nodes in the left video image: the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node. A total of 200 consecutive frames of images are selected as subsequent processing images; Obtain four three-dimensional coordinates in each of the 200 frames of images, which are the three-dimensional coordinates of the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node relative to the midpoint of the line connecting the left camera and the right camera, respectively. One three-dimensional coordinate represents a sequence point, and 200 sequence points of each of the left shoulder node, the right shoulder node, the right elbow node, and the right wrist node are obtained respectively; the 200 sequence points of the left shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain a three-dimensional original trajectory sequence of the left shoulder node; the 200 sequence points of the right shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain a three-dimensional original trajectory sequence of the right shoulder node; the 200 sequence points of the right elbow shoulder node are arranged from front to back according to the time sequence in the 200 frames of images to obtain a three-dimensional original trajectory sequence of the right elbow node; the 200 sequence points of the right wrist node are arranged from front to back according to the time sequence in the 200 frames of images to obtain a three-dimensional original trajectory sequence of the right wrist node, and a total of four three-dimensional original trajectory sequences are obtained; (3.2) Obtain the three-dimensional relative trajectory sequence of the right wrist node and the three-dimensional relative trajectory sequence of the right elbow node Set a variable q with an initial value of 1, take the horizontal projection of the line connecting the qth sequence point in the three-dimensional original trajectory sequence of the left shoulder node and the qth sequence point in the three-dimensional original trajectory sequence of the right shoulder node as the x-axis, set the vertical direction as the z-axis, and construct a right-handed Cartesian coordinate system as the qth human coordinate system according to the right-hand rule of the Cartesian coordinate system; calculate the relative coordinates of the qth sequence point in the three-dimensional original trajectory sequence of the right wrist node in the qth human coordinate system through the coordinate transformation formula, representing the qth sequence point in the three-dimensional relative trajectory sequence of the right wrist node; calculate the relative coordinates of the qth sequence point in the three-dimensional original trajectory sequence of the right elbow node in the qth human coordinate system through the coordinate transformation formula, representing the qth sequence point in the three-dimensional relative trajectory sequence of the right elbow node; Keep increasing the q value by one and repeat step (3.2) until the q value reaches 200, and obtain the three-dimensional relative trajectory sequence of the right wrist node and the three-dimensional relative trajectory sequence of the right elbow node respectively; (3.3) Obtain the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node respectively Calculate the wrist distance between adjacent sequence points in the three-dimensional relative trajectory sequence of the right wrist node, and calculate the elbow distance between adjacent sequence points in the three-dimensional relative trajectory sequence of the right elbow node; The motion speed of each sequence point in the three-dimensional relative trajectory sequence of the right wrist node is calculated according to the wrist distance. Based on the motion speed, each sequence point in the three-dimensional relative trajectory sequence of the right wrist node is classified into high-speed points and low-speed points through a clustering algorithm. The motion speed of each sequence point in the three-dimensional relative trajectory sequence of the right elbow node is calculated according to the elbow distance. Based on the motion speed, each sequence point in the three-dimensional relative trajectory sequence of the right elbow node is classified into high-speed points and low-speed points through a clustering algorithm. The continuously distributed low-speed points are set as the static phase, and the continuously distributed high-speed points are set as the moving phase. After eliminating the static phase, the three-dimensional motion trajectory sequence of the right wrist node containing only the moving phase and the three-dimensional motion trajectory sequence of the right elbow node containing only the moving phase are obtained respectively; The continuous distribution means that: more than five consecutive points in the three-dimensional relative trajectory sequence of the right elbow node are simultaneously classified as low-speed points or high-speed points; or more than five consecutive points in the three-dimensional relative trajectory sequence of the right wrist node are simultaneously classified as low-speed points or high-speed points; (4) 3D dynamic gesture recognition Based on the established 3D dynamic gesture classification template library, the specific steps for 3D dynamic gesture recognition are as follows: (4.1) Obtaining the most similar dynamic gesture The tester makes a 3D dynamic gesture to be recognized, and according to steps (2) and (3), the 3D motion trajectory sequence of the right wrist node and the 3D motion trajectory sequence of the right elbow node are obtained. Then, the 3D motion trajectory sequence of the right wrist node and the right elbow node are matched with each 3D dynamic gesture classification template in the 3D dynamic gesture classification template library by using the improved dynamic time warping algorithm. The specific steps of similarity matching are as follows: Calculate the improved dynamic time warping distance between the three-dimensional motion trajectory sequence of the right wrist node to be identified and the fine template trajectory sequence of the right wrist node in the three-dimensional dynamic gesture classification template to obtain the improved dynamic time warping distance of the right wrist node; Calculate the improved dynamic time warping distance between the three-dimensional motion trajectory sequence of the right elbow node to be identified and the fine template trajectory sequence of the right elbow node in the three-dimensional dynamic gesture classification template to obtain the improved dynamic time warping distance of the right elbow node; The average value of the improved dynamic time warping distance of the right wrist node and the improved dynamic time warping distance of the right elbow node is taken as the similarity evaluation value between the 3D motion trajectory sequence of the right wrist node or right elbow node to be identified and the 3D dynamic gesture classification template. The smaller the similarity evaluation value, the more similar it is. A 3D dynamic gesture classification template that is most similar to the 3D motion trajectory sequence of the right wrist node or the right elbow node to be identified is selected from the 3D dynamic gesture classification template library. The 3D dynamic gesture corresponding to the 3D dynamic gesture classification template is recorded as the most similar dynamic gesture, and the corresponding similarity evaluation value is called the minimum dynamic time warping distance. The improved dynamic time warping algorithm is obtained by adding a correction operation step to the dynamic time warping algorithm; (4.2) Obtaining the results of three-dimensional dynamic gesture recognition If the minimum dynamic time warping distance is less than 0.2m, the 3D dynamic gesture to be recognized is considered to be successfully recognized, and the most similar dynamic gesture is the gesture recognition result; If the minimum dynamic time warping distance is greater than 0.2m, the 3D dynamic gesture to be recognized by the tester is considered to be different from the most similar dynamic gesture, that is, the dynamic gesture made by the tester has not yet been recorded in the 3D dynamic gesture classification template library; The operations for establishing the three-dimensional dynamic gesture classification template library are as follows: Before officially starting the three-dimensional dynamic gesture recognition operation, the tester needs to continuously repeat the target three-dimensional dynamic gesture, and continuously extract the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node through steps (2) and (3). The first extracted three-dimensional motion trajectory sequence of the right wrist node is set as the rough template trajectory sequence of the right wrist node, and the first extracted three-dimensional motion trajectory sequence of the right elbow node is set as the rough template trajectory sequence of the right elbow node; in the process of the tester continuously repeating the target three-dimensional dynamic gesture, the upper computer continuously synthesizes and replaces the two rough template trajectory sequences through the improved dynamic time warping algorithm and template synthesis algorithm, and completes 200 template synthesis and replacement. The rough template trajectory sequence of the right wrist node after replacement is called the fine template trajectory sequence of the right wrist node, and the rough template trajectory sequence of the right elbow node after completing 200 template synthesis and replacement is called the fine template trajectory sequence of the right elbow node. The two fine template trajectory sequences are collectively called the three-dimensional dynamic gesture classification template of the target three-dimensional dynamic gesture; new target three-dimensional dynamic gestures are continuously designed and three-dimensional dynamic gesture classification templates are obtained until the number of target three-dimensional dynamic gestures reaches four or more, that is, more than four three-dimensional dynamic gesture classification templates are obtained. More than four three-dimensional dynamic gesture classification templates constitute a three-dimensional dynamic gesture classification template library, which is used as the basis for gesture recognition when formally performing three-dimensional dynamic gesture recognition operations.

2. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 1, characterized in that: The distance between the lens of the left camera and the lens of the right camera is 120 mm; The field of view of the binocular depth camera is 110° horizontally × 70° vertically × 120° diagonally.

3. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 1, characterized in that: In step (1.3), the large convex hull defect refers to a convex hull defect with a concave depth exceeding one-quarter of the length of the palm.

4. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 1, characterized in that: In step (3.2), the coordinate transformation formula is a formula for calculating the three-dimensional relative coordinates of the qth sequence point in the three-dimensional relative trajectory sequence of the right elbow node or the qth sequence point in the three-dimensional relative trajectory sequence of the right wrist node relative to the qth human body relative coordinate system. The specific formula is as follows: In formula (1)(2)(3)(4), (x A ,y A , z A ) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the left shoulder node, (x B ,y B , z B ) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the right shoulder node, (x, y, z) is the absolute three-dimensional coordinate of the qth sequence point in the three-dimensional original trajectory sequence of the right elbow node or the qth sequence point in the three-dimensional original trajectory sequence of the right wrist node that requires coordinate transformation, (x′, y′, z′) is the three-dimensional relative coordinate of the qth sequence point in the three-dimensional relative trajectory sequence of the right elbow node or the qth sequence point in the three-dimensional relative trajectory sequence of the right wrist node that requires relative coordinate calculation in the qth human body relative coordinate system, and the unit of all coordinates is m.

5. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 1, characterized in that: In step (3.3), all sequence points in the three-dimensional relative trajectory sequence are divided into two types: low-speed points and high-speed points through a clustering algorithm. The specific operation steps are as follows: (3.3.1) Calculate the velocity of all sequence points in the three-dimensional relative trajectory sequence. Randomly select two sequence points a and b in the three-dimensional relative trajectory sequence. Calculate the difference between the velocity of all sequence points in the entire sequence trajectory and the velocity of sequence points a and b. If the velocity difference between a sequence point and sequence point a is less than the velocity difference between a sequence point and sequence point b, then classify the sequence point as Class A. If the velocity difference between a sequence point and sequence point a is greater than the velocity difference between a sequence point and sequence point b, then classify the sequence point as Class B. (3.3.2) Calculate the average speed of A for class A and the average speed of B for class B, find the sequence point in class A whose speed is closest to the average speed of A, and use this sequence point to replace the original sequence point a; find the sequence point in class B whose speed is closest to the average speed of B, and use this sequence point to replace the original sequence point b; repeat step (3.3.1) once to reclassify all sequence points; (3.3.3) Repeat step (3.3.2) at least twice. After each repetition, check whether sequence point a and sequence point b are replaced. If sequence point a and sequence point b are no longer replaced with the repetition of step (3.3.2), it means that the optimal classification has been achieved. Stop repeating step (3.3.2) and classify the class with the larger average speed between class A and class B as the high-speed class. All sequence points in the high-speed class are called high-speed points. The class with the smaller average speed is called the low-speed class. All sequence points in the low-speed class are called low-speed points.

6. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 1, characterized in that: In step (4), the steps for establishing the three-dimensional dynamic gesture classification template library are as follows: (A) Obtaining transition trajectory sequence The tester repeats the target three-dimensional dynamic gesture once, and extracts the three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node through steps (2) and (3). The three-dimensional motion trajectory sequence of the right wrist node and the three-dimensional motion trajectory sequence of the right elbow node are respectively matched with the recorded rough template trajectory sequences of the right wrist node and the right elbow node through the improved dynamic time warping algorithm for similarity, and obtains the improved dynamic time warping distance of the right wrist node between the three-dimensional motion trajectory sequence of the right wrist node and the rough template trajectory sequence of the right wrist node, and the improved dynamic time warping distance of the right elbow node between the three-dimensional motion trajectory sequence of the right elbow node and the rough template trajectory sequence of the right elbow node, thus completing a similarity calculation; The improved dynamic time warping distance of the right wrist node and the improved dynamic time warping distance of the right elbow node respectively represent the similarity between the three-dimensional motion trajectory sequences of the two joint points and the rough template trajectory sequence. If the obtained improved dynamic time warping distances of the right wrist node or the right elbow node are both less than 0.2m, it is considered that the three-dimensional motion trajectory sequences of the two joint points are sufficiently similar to the recorded rough template trajectory sequence; the three-dimensional motion trajectory sequences of the two joint points are called the transition trajectory sequences of the two joint points, namely the right wrist node transition trajectory sequence and the right elbow node transition trajectory sequence; If one of the obtained improved dynamic time warping distances of the right wrist node and the improved dynamic time warping distances of the right elbow node is greater than 0.2m or both are greater than 0.2m, repeat the similarity calculation until both improved dynamic time warping distances are less than 0.2m. The three-dimensional motion trajectory sequences of the two joints are called the transition trajectory sequences of the two joints, i.e., the right wrist node transition trajectory sequence and the right elbow node transition trajectory sequence. (B) Obtaining a fine template trajectory sequence The right wrist node transition trajectory sequence and the right wrist node rough template trajectory sequence are synthesized by the template synthesis algorithm to obtain a new right wrist node rough template trajectory sequence; the right elbow node transition trajectory sequence and the right elbow node rough template trajectory sequence are synthesized to obtain a new right elbow node rough template trajectory sequence; and the previous rough template trajectory sequences are replaced and synthesized again. After accumulating 200 synthesis and replacement, the right wrist node fine template trajectory sequence and the right elbow node fine template trajectory sequence are obtained respectively. (C) Obtaining a 3D dynamic gesture classification template library Record the fine template trajectory sequence of the right wrist node and the fine template trajectory sequence of the right elbow node as a set of fine template trajectory sequences, which is called the first 3D dynamic gesture classification template. The host computer stores the first 3D dynamic gesture classification template in a blank file, which is called the 3D dynamic gesture classification template library. Assume that an unrecorded three-dimensional dynamic gesture is used as the second target three-dimensional dynamic gesture. A second three-dimensional dynamic gesture classification template of the second three-dimensional dynamic gesture is obtained through steps (A) and (B), and stored in the three-dimensional dynamic gesture classification template library. New target three-dimensional dynamic gestures are repeatedly designed, and several three-dimensional dynamic gesture classification templates are obtained and stored in the three-dimensional dynamic gesture classification template library until the number of target three-dimensional dynamic gestures recorded in the three-dimensional dynamic gesture classification template library is greater than four, thereby obtaining a three-dimensional dynamic gesture classification template library that can be used as a basis for three-dimensional dynamic gesture classification.

7. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 6, characterized in that: In step (A), the steps for performing similarity matching using the improved dynamic time warping algorithm are as follows: The right elbow node similarity matching operation is as follows: (A1) Construct an m×n matrix, called a distance matrix; the element in the i-th row and j-th column of the distance matrix is the Euclidean distance between the i-th sequence point in the three-dimensional motion trajectory sequence of the right elbow node and the j-th sequence point in the rough template trajectory sequence of the right elbow node; The m in the distance matrix is the number of sequence points contained in the three-dimensional motion trajectory sequence of the right elbow node; n in the distance matrix is the number of sequence points contained in the rough template trajectory sequence of the right elbow node; (A2) Then construct an m×n matrix, called the cumulative distance matrix. The element in the i-th row and j-th column of the cumulative distance matrix is obtained by the recursive formula (5): D ij =d ij +min{D i，j-1 ,D i-1，j-1 ,D i-1,j }——(5) In formula (5), d ij is the element in row i and column j of the distance matrix, D ij To accumulate the elements in row i and column j of the distance matrix, when row number i or column number j is less than 1, 0 is used instead of D ij , D 11 =d 11 , starting from the element in the first row and first column, calculate all the elements in the cumulative distance matrix from the first row to the last row through the recursive formula (5), then the element D in the mth row and nth column is mn The dynamic time warping distance calculated by the traditional dynamic time warping algorithm. The dynamic time warping distance value will increase with the increase of m and n, so it needs to be corrected. (A3) The correction steps are as follows: In order to correct the influence of the size of m and n on the traditional dynamic time warping distance, from D mn First, recursively follow the recursive formula (5) to find and record all the distance matrices in step (A1) that participated in D mn Calculate the element d ij , these elements d ij The set of is called the minimum warping path between the two trajectory sequences; the elements in the minimum warping path in each column of the distance matrix are averaged, the average values of each column are accumulated and divided by the total number of columns n to obtain the improved dynamic time warping distance between the three-dimensional motion trajectory sequence of the right elbow node and the rough template trajectory sequence of the right elbow node; the smaller the improved dynamic time warping distance, the higher the similarity; The similarity matching operation of the right wrist node is the same as that of the right elbow node; That is, the improved dynamic time warping distance of the right elbow node and the improved dynamic time warping distance of the right wrist node are obtained; If both improved dynamic time warping distances are less than 0.2m, the new 3D motion trajectory sequences of the two joints are considered sufficiently similar to the recorded rough template trajectory sequence. If one or both of the two improved dynamic time warping distances are greater than 0.2m, repeat step (A) until both improved dynamic time warping distances are less than 0.2m; thus, the transition trajectory sequences of the right elbow node and the right wrist node are obtained respectively. In step (B), the template synthesis algorithm is used to perform 200 template synthesis and replacement operations cumulatively. The template synthesis and replacement operations for the right elbow node are as follows: (B1) Obtain two weight values The template is synthesized by the rough template trajectory sequence of the right elbow node and the transition trajectory sequence of the right elbow node. The weight value of the rough template trajectory sequence of the right elbow node is set to e, and the weight value of the transition trajectory sequence of the right elbow node is set to f. The 3D motion trajectory sequence of the right elbow node recorded when the tester first performs the target 3D dynamic gesture is used as the initial rough template trajectory sequence of the right elbow node. The weight value e is recorded as 1. After that, the weight value e of the rough template trajectory sequence of the right elbow node is increased by 1 each time the template synthesis is performed. The weight value f of the right elbow node transition trajectory sequence in template synthesis is always 1; (B2) Obtaining the incremental sequence points Repeat step (A) once to obtain a right elbow node transition trajectory sequence, and record the minimum regularized path between the right elbow node transition trajectory sequence generated in the correction operation of step (A3) and the right elbow node rough template trajectory sequence; Assume that the number of Euclidean distance values contained in the minimum regularized path is u; each of the u Euclidean distance values is obtained by calculating the Euclidean distance between a sequence point in the transition trajectory sequence of the right elbow node and a sequence point that matches the rough template trajectory sequence of the right elbow node. The two sequence points used to calculate each Euclidean distance value are recorded as a matching point pair for each Euclidean distance value, and the u Euclidean distance values correspond to u matching point pairs respectively. u Euclidean distance values are respectively located in the distance matrix of step (A1), and the row number and column number of each Euclidean distance value in the distance matrix are recorded. The row number of each Euclidean distance value in the distance matrix is set as the row value of the matching point pair of each Euclidean distance value, and the column number of each Euclidean distance value in the distance matrix is set as the column value of the matching point pair of each Euclidean distance value; For each of the u matching point pairs, calculate the weighted average coordinate of each matching point pair based on the two 3D coordinates of each matching point pair according to the weight values e and f recorded in step (B1), obtaining u weighted average coordinates; set each weighted average coordinate as the 3D coordinate of an incremental sequence point, obtaining u incremental sequence points; According to the correspondence between the u point-increasing sequence points and the u matching points, the row value of each matching point pair is set to the row value of the corresponding point-increasing sequence point, and the column value of each matching point pair is set to the column value of the corresponding point-increasing sequence point, to obtain u point-increasing sequence points with row values and column values; (B3) Single-point template synthesis and replacement Sort u increment sequence points with row values and column values, arranging increment sequence points with smaller row values before increment sequence points with larger row values, and arranging increment sequence points with smaller column values before increment sequence points with larger column values, to obtain sorted u increment sequence points; all sorted u increment sequence points together constitute a trajectory sequence, which is set as the increment trajectory sequence; Delete the right elbow node rough template trajectory sequence, set the added point trajectory sequence to the right elbow node rough template trajectory sequence with the number of synthesis increased by one, complete one point-added template synthesis and replacement, and obtain the right elbow node rough template trajectory sequence with the number of times of participation in synthesis increased by one, that is, the right elbow node rough template trajectory sequence with the weight value e increased by one; (B4) Complete 100 times of template synthesis Repeat steps (B1), (B2), and (B3) to complete one hundred times of template synthesis and replacement, and obtain a rough template trajectory sequence of the right elbow node with one hundred synthesis times, and the weight value e reaches 101; (B5) Obtain constant sequence points Repeat step (A) once to obtain a right elbow node transition trajectory sequence, and record the minimum regularized path between the right elbow node transition trajectory sequence generated in the correction operation of step (A3) and the right elbow node rough template trajectory sequence; Assume that the number of Euclidean distance values contained in the minimum regularized path is u; each of the u Euclidean distance values is obtained by calculating the Euclidean distance between a sequence point in the transition trajectory sequence of the right elbow node and a sequence point that matches the rough template trajectory sequence of the right elbow node, and the two sequence points used to calculate each Euclidean distance value are recorded as a matching point pair for each Euclidean distance value, obtaining u matching point pairs; For each of the u matching point pairs, calculate the weighted average coordinate of each matching point pair based on the two 3D coordinates of each matching point pair according to the weight value e and the weight value f recorded in step (B1), thereby obtaining u weighted average coordinates; set each weighted average coordinate as the 3D coordinate of a temporary sequence point, thereby obtaining u temporary sequence points; the u temporary sequence points correspond to the u Euclidean distance values; The u Euclidean distance values are respectively located in the distance matrix of step (A1), and the u Euclidean distance values obtained in step (B3) are divided into n groups according to the column number of each Euclidean distance value in the distance matrix. The Euclidean distance values located in the same column of the distance matrix are classified into the same group, and the group number is the same as the column number; the distance matrix described in step (A1) has a total of n columns, and the n columns in the distance matrix correspond to the n groups in the u temporary sequence points; Divide u temporary sequence points into n groups, where the u temporary sequence point groups are the same as the corresponding u Euclidean distance value groups; Calculate the average coordinates of the three-dimensional coordinates of each group of temporary sequence points in n groups of temporary sequence points, set the group number of each group of temporary sequence points to the group value of the calculated average coordinates, and obtain n average coordinates with group values; set each average coordinate of the n average coordinates with group values to the three-dimensional coordinates of a constant point sequence point, and set the group value of each average coordinate to the group value of the corresponding constant point sequence point, and obtain n constant point sequence points with group values; (B6) Single constant point template synthesis and replacement Sort n constant point sequence points with group values, and place the constant point sequence points with smaller group values before the constant point sequence points with larger group values, to obtain n sorted constant point sequence points; all the sorted n constant point sequence points together constitute a trajectory sequence, which is set as the constant point trajectory sequence; Delete the right elbow node rough template trajectory sequence with more than 100 synthesis times, set the constant point trajectory sequence to the right elbow node rough template trajectory sequence with more than 100 synthesis times plus 1, complete one constant point template synthesis and replacement, and obtain the right elbow node rough template trajectory sequence with more than 100 synthesis times plus 1, that is, the right elbow node rough template trajectory sequence with a weight value e of more than 101 plus 1; (B7) Obtain the fine template trajectory sequence of the right elbow node Repeat steps (B1), (B5), and (B6) to complete 100 constant point template synthesis and replacements, and obtain a rough template trajectory sequence of the right elbow node synthesized 200 times. The weight value e reaches 201. The rough template trajectory sequence of the right elbow node synthesized 200 times is set as the fine template trajectory sequence of the right elbow node, and the fine template trajectory sequence of the right elbow node is obtained. The template synthesis and replacement operations of the right wrist node are the same as those of the right elbow node, and a fine template trajectory sequence of the right wrist node is obtained.

8. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 7, characterized in that: In step (B2) or step (B5), the calculation operation of calculating the weighted average coordinates of each matching point pair based on the two three-dimensional coordinates of each matching point pair is as follows: Obtain the first three-dimensional coordinates of the sequence point in the matching point pair from the rough template trajectory sequence of the right elbow node, and multiply the values of the three dimensions of the first three-dimensional coordinates by the weight value e respectively to obtain the first weighted coordinates; Obtain the second three-dimensional coordinates of the sequence point in the right elbow node transition trajectory sequence from the matching point pair, and multiply the values of the three dimensions of the second three-dimensional coordinates by the weight value f to obtain a second weighted coordinate; The x-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the x-dimensional value in the weighted average coordinate of the matching point pair; the y-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the y-dimensional value in the weighted average coordinate of the matching point pair; the z-dimensional values of the first weighted coordinate and the second weighted coordinate are averaged as the z-dimensional value in the weighted average coordinate of the matching point pair; the weighted average coordinate of the matching point pair is obtained according to the x-dimensional value, the y-dimensional value and the z-dimensional value.

9. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 7, characterized in that: In step (B5), the calculation operation of the average coordinates of the three-dimensional coordinates of each group of temporary sequence points is as follows: Sum the x-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the x-dimension value of the average coordinates. Sum the y-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the y-dimension value of the average coordinates. Sum the z-dimension values of the three-dimensional coordinates of all temporary sequence points in the same group, and then divide it by the total number of temporary sequence points in this group to obtain the z-dimension value of the average coordinates. The average coordinates of the three-dimensional coordinates of all temporary sequence points in the same group are obtained according to the x-dimension value, the y-dimension value and the z-dimension value.

10. The three-dimensional dynamic hand gesture recognition method based on the improved dynamic time warping algorithm according to claim 1, characterized in that: In step (4.1), the improved dynamic time warping algorithm is obtained by adding a correction operation step to the dynamic time warping algorithm. The specific process of the correction operation step is the same as the correction operation step of the improved dynamic time warping algorithm used in the establishment operation of the three-dimensional dynamic gesture classification template library.

Images