Model training method, gesture recognition method, device, apparatus and medium

Abstract

This application discloses a method, apparatus, device, and medium for training a joint rotation detection model, relating to the field of image processing technology. The method includes: acquiring human hand image data and extracting first hand key points; determining the first rotation axis angle of each joint point using inverse dynamics; inputting the human hand image data into an initial joint rotation detection model to output a six-dimensional rotation vector, and obtaining the second rotation axis angle corresponding to the six-dimensional rotation vector; determining the second hand key points based on the first hand key points and the rotation matrix; constructing loss functions based on the first and second rotation axis angles and the first and second hand key points respectively; and training the initial joint rotation detection model using the overall loss function to obtain the joint rotation detection model. By training the model using loss functions for the joint rotation axis angles and hand key points, the stability and performance of the model can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a model training method, a gesture recognition method, a device, equipment, and a medium. Background Technology

[0002] With the development of technology, smart devices have gradually entered people's daily lives, such as smartphones with cameras, smart TVs, and car central control systems. Among various technologies, image-based gesture recognition technology is widely used. Without text or voice recording devices, gestures can help express meaning. Currently, there are three main methods for gesture recognition based on deep learning models: (1) Image classification to recognize gestures. The advantage of this method is that the whole process is simple and time-consuming. However, the disadvantage is that it is difficult to add new gestures. Each time a gesture is added, the entire image model needs to be retrained. In addition, the size of the hand in the image has a significant impact on the recognition accuracy. (2) Detecting key points of the hand and adding gesture classification. This method does not require retraining the image model each time. It only requires retraining the classification model, which enhances scalability. However, when the hand rotates as a whole, the gesture classification model will be difficult to train because the key points change as a whole. (3) Detecting key points of the hand and performing inverse dynamics before performing gesture classification. This method can overcome the problems in the first two methods. However, since the inverse dynamics process increases in time with the increase of key points and the solution result is not unique, it is easy to cause ambiguity in classification.

[0003] Existing technologies propose a method to drive virtual human movement by acquiring a gesture motion database composed of human upper limb and hand details using traditional digital image processing methods. The approach involves first extracting contours and textures from images and then synthesizing them to create a gesture image database; then, pre-creating the motion sequences of all human skeleton models corresponding to the gesture database, with each gesture corresponding to a skeleton model motion sequence. However, this approach primarily relies on image recognition methods for gesture classification, focusing on how to construct human morphemes through gestures to achieve a strong correspondence between virtual human motion sequences and gestures, rather than how to recognize gestures. Furthermore, the motion parameters used in the mentioned forward kinematic representation are artificially pre-created motion parameters, not derived from real gesture images.

[0004] In summary, how to take into account the rotation of human hand joints during gesture recognition, and avoid inaccurate gesture recognition caused by direct key point detection or differences in human hand size, is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a model training method, gesture recognition method, device, equipment, and medium. By first detecting the rotation of human hand joints using a joint rotation detection model, and then performing subsequent gesture recognition, it can avoid inaccuracies in gesture recognition caused by directly classifying gestures based on keypoint detection or by hand size. Furthermore, by training the model using a loss function based on the joint rotation axis angle and a loss function based on the hand keypoints, the stability and performance of the model can be improved. The specific solution is as follows:

[0006] Firstly, this application discloses a method for training a joint rotation detection model, including:

[0007] The process involves acquiring human hand image data, extracting key points from the human hand image data to obtain first hand key points, and processing the human hand image data based on the first hand key points using inverse dynamics to obtain the first rotation axis angle of each joint of the human hand.

[0008] The hand image data is input into the initial joint rotation detection model to output a six-dimensional rotation vector, and the second rotation axis angle corresponding to the six-dimensional rotation vector is obtained.

[0009] The rotation matrix is ​​determined using the second rotation axis angle, and the second hand key point is determined based on the first hand key point and the rotation matrix;

[0010] A first loss function is constructed based on the first rotation axis angle and the second rotation axis angle, and a second loss function is constructed based on the first hand key point and the second hand key point;

[0011] The initial joint rotation detection model is trained using the first loss function and the second loss function to obtain the trained joint rotation detection model.

[0012] Optionally, acquiring the hand image data includes:

[0013] Acquire human hand image data using a 3D scanner or RGB-D camera.

[0014] Optionally, the step of extracting key points from the hand image data to obtain first hand key points includes:

[0015] Extract the first spatial coordinates of the hand joints and the second spatial coordinates of the fingertips from the hand image data;

[0016] The first key hand point is obtained based on the first spatial coordinate and the second spatial coordinate.

[0017] Optionally, obtaining the second rotation axis angle corresponding to the six-dimensional rotation vector includes:

[0018] The six-dimensional rotation vector is subjected to Gram-Schmidt orthogonalization to obtain the corresponding second rotation axis angle.

[0019] Optionally, determining the rotation matrix using the second rotation axis angle includes:

[0020] The rotation matrix is ​​determined based on the rotation axis and rotation angle in the second rotation axis angle, and using the Rodriguez rotation formula.

[0021] Optionally, the process of processing the human hand image data based on the first hand key points using the inverse dynamics method to obtain the first rotation axis angle of each joint of the human hand further includes:

[0022] Predetermine the initial posture of the standard human hand;

[0023] Based on the initial posture, a first rotation axis angle corresponding to the human hand image data is determined.

[0024] Secondly, this application discloses a gesture recognition method, characterized in that it includes:

[0025] A target hand image to be identified is obtained, and the target hand image is processed using the joint rotation detection model trained by the aforementioned disclosed joint rotation detection model training method to obtain a set of rotation axis angles of the hand joints.

[0026] The set of rotation axis angles is input into a preset gesture classification model to obtain gesture recognition results.

[0027] Thirdly, this application discloses a joint rotation detection model training device, comprising:

[0028] The image data processing module is used to acquire human hand image data, extract key points from the human hand image data to obtain first hand key points, and process the human hand image data based on the first hand key points using the inverse dynamics method to obtain the first rotation axis angle of each joint of the human hand.

[0029] The rotation axis angle acquisition module is used to input the human hand image data into the initial joint point rotation detection model to output a six-dimensional rotation vector, and to acquire the second rotation axis angle corresponding to the six-dimensional rotation vector;

[0030] The hand key point determination module is used to determine the rotation matrix using the second rotation axis angle, and to determine the second hand key point based on the first hand key point and the rotation matrix;

[0031] The loss function construction module is used to construct a first loss function based on the first rotation axis angle and the second rotation axis angle, and to construct a second loss function based on the first hand key point and the second hand key point;

[0032] The model training module is used to train the initial joint rotation detection model using the first loss function and the second loss function to obtain the trained joint rotation detection model.

[0033] Fourthly, this application discloses an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is used to execute the computer program to implement the steps of the aforementioned disclosed joint rotation detection model training method.

[0036] Fifthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed joint rotation detection model training method.

[0037] As can be seen, this application acquires human hand image data, extracts key points from the human hand image data to obtain first hand key points, and processes the human hand image data using inverse dynamics based on the first hand key points to obtain first rotation axis angles of each joint of the human hand; inputs the human hand image data into an initial joint rotation detection model to output a six-dimensional rotation vector, and obtains a second rotation axis angle corresponding to the six-dimensional rotation vector; uses the second rotation axis angle to determine a rotation matrix, and determines second hand key points based on the first hand key points and the rotation matrix; constructs a first loss function based on the first rotation axis angle and the second rotation axis angle, and constructs a second loss function based on the first hand key points and the second hand key points; trains the initial joint rotation detection model using the first loss function and the second loss function to obtain a trained joint rotation detection model. Therefore, this application first extracts key points from the acquired hand image data to obtain first hand key points. Based on the first hand key points, the first rotation axis angle of each joint of the hand is determined using inverse dynamics. The hand image data is input into an initial joint rotation detection model to output a six-dimensional rotation vector, which is then transformed into a corresponding second rotation axis angle. The rotation matrix determined by the second rotation axis angle and the first hand key points are then used to obtain the second hand key points. Corresponding loss functions are then constructed based on the first and second hand key points, as well as the first and second rotation axis angles, respectively. The initial joint rotation detection model is then trained using the overall loss function to obtain a trained joint rotation detection model. After extracting key points from the acquired hand image data, the above scheme further requires obtaining the rotation axis angle of the hand joints using inverse dynamics. In other words, by emphasizing the rotation between each joint point, this application avoids the problems of direct key point detection in subsequent gesture recognition and the low recognition accuracy caused by changes in hand size in pure image recognition. In addition, by training the model by fusing the loss function of the joint rotation axis angle and the loss function of the hand key points, the model constraints can be increased, thereby accelerating the convergence of the model and improving the stability and performance of the model. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 This is a flowchart of a joint rotation detection model training method disclosed in this application;

[0040] Figure 2 This is a schematic diagram of a specific joint rotation detection model structure disclosed in this application;

[0041] Figure 3 This is a flowchart of a specific joint rotation detection model training method disclosed in this application;

[0042] Figure 4 This is a flowchart of a gesture recognition method disclosed in this application;

[0043] Figure 5 This is a schematic diagram of a joint rotation detection model training device disclosed in this application.

[0044] Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0046] Currently, there are three main methods for gesture recognition based on deep learning models: (1) Image classification to recognize gestures. The advantage of this method is that the whole process is simple and time-consuming. However, the disadvantage is that it is difficult to add new gestures. Each time a gesture is added, the entire image model needs to be retrained. In addition, the size of the hand in the image has a significant impact on the recognition accuracy. (2) Detecting key points of the hand and adding gesture classification. This method does not require retraining the image model each time. It only requires retraining the classification model, which enhances scalability. However, when the hand rotates as a whole, the gesture classification model will be difficult to train because the key points change as a whole. (3) Detecting key points of the hand and performing inverse dynamics before performing gesture classification. This method can overcome the problems in the first two methods. However, the inverse dynamics process will increase the time consumption with the increase of key points, and the solution result is not unique, which can easily cause ambiguity in classification. To address this, this application discloses a model training method, gesture recognition method, device, equipment, and medium. By first detecting the rotation of human hand joints using a joint rotation detection model and then performing subsequent gesture recognition, it avoids inaccurate gesture recognition caused by directly classifying gestures based on key point detection or by the size of the human hand. Furthermore, by training the model using a loss function that constructs the joint rotation axis angle and a loss function that constructs the hand key points, the stability and performance of the model can be improved.

[0047] See Figure 1 As shown in the embodiment of this application, a method for training a joint rotation detection model is disclosed, the method comprising:

[0048] Step S11: Acquire human hand image data, extract key points from the human hand image data to obtain first hand key points, and process the human hand image data using inverse dynamics based on the first hand key points to obtain the first rotation axis angle of each joint of the human hand.

[0049] In this embodiment, hand image data is first acquired. In a specific implementation, open-source 3D hand image datasets such as FreiHand and InterHand can be used directly, and the image type can be RGB images. Then, keypoint extraction processing is performed on the acquired hand image data to obtain 3D hand keypoints, i.e., the first hand keypoints. Based on the first hand keypoints, the hand image data is then processed using inverse dynamics to obtain the first rotation axis angle between each joint of the hand. It should be noted that inverse kinematics is a method that determines the entire skeletal chain by first determining the position of the child bone and then inversely deriving the position of the n-level parent bone on the skeletal chain. In contrast, forward kinematics, also known as forward dynamics, refers to a hierarchical structure that completely follows the parent-child relationship, with the parent level driving the movement of the child level. In addition, the angle (AngleAxis) is a rotation angle defined by a unit vector plus a rotation angle defined by a scalar, usually represented as [x, y, z, theta]. The first three parameters represent the rotation axis, and the last parameter represents the rotation angle.

[0050] Additionally, it should be noted that in the acquired hand image data, some images may contain other objects besides the hand. In this case, target detection is needed to determine the largest target in the image, that is, to determine the range of the hand, which is convenient for subsequent key point extraction and acquisition of the rotation axis angle of each joint.

[0051] Step S12: Input the human hand image data into the initial joint rotation detection model to output a six-dimensional rotation vector, and obtain the second rotation axis angle corresponding to the six-dimensional rotation vector.

[0052] In this embodiment, human hand image data is input into an initial joint rotation detection model to output a six-dimensional rotation vector. For details, please refer to [reference needed]. Figure 2 As shown, Figure 2This diagram illustrates a specific joint rotation detection model structure disclosed in this application. First, an RGB image of a human hand is input. The image is then processed sequentially through a backbone network, global average pooling, and tensor transformation to output a corresponding six-dimensional rotation vector. After obtaining the six-dimensional rotation vector, the corresponding second rotation axis angle is obtained, that is, the six-dimensional rotation vector representation A is transformed into a rotation axis angle representation B. In other words, this embodiment obtains the rotation axis angle of the human hand joints using a deep learning model.

[0053] Step S13: Determine the rotation matrix using the second rotation axis angle, and determine the second hand key point based on the first hand key point and the rotation matrix.

[0054] In this embodiment, the rotation matrix is ​​determined using the obtained second rotation axis angle, and then the second hand key points are determined based on the first hand key points and the rotation matrix. In a specific implementation, the spatial coordinates of the extracted M first hand key points are denoted as P = {P1, P2, ... P...}. M Let the rotation matrix be R, then the corresponding second hand keypoint is determined as follows: P i '=P i-1 +R*(P i -P i-1 ), where P i ' represents the spatial coordinates of the i-th key point in the second hand key points, P i and P i-1 Let be the spatial coordinates of the i-th and (i-1)-th key points in the first hand key points.

[0055] Step S14: Construct a first loss function based on the first rotation axis angle and the second rotation axis angle, and construct a second loss function based on the first hand key point and the second hand key point.

[0056] In this embodiment, a first loss function is constructed based on the first rotation axis angle and the second rotation axis angle to represent the loss between the independent rotation of each of the n joints and the real data. The first rotation axis angle is the real data extracted directly from the human hand image data, denoted as B. gt The second rotation axis angle is the data predicted by the model, denoted as B. pred And the first loss function is denoted as L. Joints The expression for the first loss function is then as follows:

[0057]

[0058] Where N is the total number of joints in the human hand. and These are the first rotation axis angle and the second rotation axis angle corresponding to the i-th joint point, respectively.

[0059] Furthermore, a second loss function is constructed based on the first and second hand keypoints to represent the overall spatial offset loss of the keypoints, i.e., used to evaluate the spatial offset between the model's results and the real data. The main purpose of adding this loss is to increase the constraint on the rigid hand model, since the rotation of each joint is relatively independent of other joints. Here, the first hand keypoint is the real data extracted directly from the hand image data, denoted as P. gt The second key point of the hand is the data predicted by the model, denoted as P. pred And the second loss function is denoted as L KinematicsShift The expression for the constructed second loss function is as follows:

[0060]

[0061] Where M represents the total number of key points on the hand. and These are the i-th first hand key point and the i-th second hand key point, respectively.

[0062] Step S15: Train the initial joint rotation detection model using the first loss function and the second loss function to obtain the trained joint rotation detection model.

[0063] In this embodiment, the initial joint rotation detection model is trained using both the first and second loss functions to obtain the trained joint rotation detection model. That is, a total loss function is constructed based on the first and second loss functions, and then the initial joint rotation detection model is trained using the total loss function. By fusing the relatively independent rotation loss of joints with the overall spatial offset loss of keypoints to train the model, model constraints are increased, thereby accelerating model convergence and improving model stability and performance.

[0064] Wherein, the total loss function L ALL The expression is:

[0065] L ALL =L Joints +L KinematicsShift .

[0066] As can be seen, this application acquires human hand image data, extracts key points from the human hand image data to obtain first hand key points, and processes the human hand image data using inverse dynamics based on the first hand key points to obtain first rotation axis angles of each joint of the human hand; inputs the human hand image data into an initial joint rotation detection model to output a six-dimensional rotation vector, and obtains a second rotation axis angle corresponding to the six-dimensional rotation vector; uses the second rotation axis angle to determine a rotation matrix, and determines second hand key points based on the first hand key points and the rotation matrix; constructs a first loss function based on the first rotation axis angle and the second rotation axis angle, and constructs a second loss function based on the first hand key points and the second hand key points; trains the initial joint rotation detection model using the first loss function and the second loss function to obtain a trained joint rotation detection model. Therefore, this application first extracts key points from the acquired hand image data to obtain first hand key points. Based on the first hand key points, the first rotation axis angle of each joint of the hand is determined using inverse dynamics. The hand image data is input into an initial joint rotation detection model to output a six-dimensional rotation vector, which is then transformed into a corresponding second rotation axis angle. The rotation matrix determined by the second rotation axis angle and the first hand key points are then used to obtain the second hand key points. Corresponding loss functions are then constructed based on the first and second hand key points, as well as the first and second rotation axis angles, respectively. The initial joint rotation detection model is then trained using the overall loss function to obtain a trained joint rotation detection model. After extracting key points from the acquired hand image data, the above scheme further requires obtaining the rotation axis angle of the hand joints using inverse dynamics. In other words, by emphasizing the rotation between each joint point, this application avoids the problems of direct key point detection in subsequent gesture recognition and the low recognition accuracy caused by changes in hand size in pure image recognition. In addition, by training the model by fusing the loss function of the joint rotation axis angle and the loss function of the hand key points, the model constraints can be increased, thereby accelerating the convergence of the model and improving the stability and performance of the model.

[0067] See Figure 3 As shown in the illustration, this application discloses a specific method for training a joint rotation detection model. Compared to the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0068] Step S21: Acquire human hand image data using a 3D scanner or RGB-D camera, and extract the first spatial coordinates of the hand joints and the second spatial coordinates of the fingertips from the human hand image data.

[0069] In this embodiment, the hand image data can be real motion information acquired using a 3D scanner or an RGB-D camera. Specifically, key point extraction involves extracting the first spatial coordinates of N hand joints and the second spatial coordinates of K fingertips from the hand image data.

[0070] Step S22: Based on the first spatial position coordinates and the second spatial position coordinates, obtain the first hand key points, and based on the first hand key points, process the human hand image data using the inverse dynamics method to obtain the first rotation axis angle of each joint of the human hand.

[0071] In this embodiment, the first hand key points are obtained based on the first and second spatial coordinates obtained above. Further, the process of processing the hand image data using inverse dynamics based on the first hand key points to obtain the first rotation axis angles of each joint of the hand also includes: pre-defining an initial posture of a standard hand; and determining the first rotation axis angle corresponding to the hand image data based on the initial posture. It should be noted that in this embodiment, the developers need to pre-define the initial posture of a standard hand or rigid hand, which is equivalent to providing an initial value for reference. The rotation axis angles of each subsequent joint are determined accordingly based on the initial posture.

[0072] Step S23: Input the human hand image data into the initial joint rotation detection model to output a six-dimensional rotation vector, and perform Gram-Schmidt orthogonalization on the six-dimensional rotation vector to obtain the corresponding second rotation axis angle.

[0073] In this embodiment, after outputting the six-dimensional rotation vector, Gram-Schmidt orthogonalization is used to transform the six-dimensional rotation vector to obtain the corresponding second rotation axis angle. The specific process is as follows:

[0074] Here, the six-dimensional rotation vector is denoted as A, and the transformed second rotation axis angle is denoted as B:

[0075]

[0076]

[0077] Therefore, the conversion formula can be expressed as:

[0078]

[0079] Step S24: Based on the rotation axis and rotation angle in the second rotation axis angle, and using the Rodriguez rotation formula, determine the rotation matrix, and based on the first hand key point and the rotation matrix, determine the second hand key point.

[0080] In this embodiment, based on the rotation axis and rotation angle in the second rotation axis angle, and using the Rodriguez rotation formula, the rotation matrix is ​​determined. Let n be the vector corresponding to the rotation axis, and θ be the rotation angle. Then, the expression for calculating the rotation matrix R is:

[0081] R = cosθ * I + (1 - cosθ) * n * n T +sinθ*Skew(n);

[0082] Where I represents the identity matrix and Skew(n) represents the antisymmetric matrix of n.

[0083] Step S25: Construct a first loss function based on the first rotation axis angle and the second rotation axis angle, and construct a second loss function based on the first hand key point and the second hand key point.

[0084] Step S26: Train the initial joint rotation detection model using the first loss function and the second loss function to obtain the trained joint rotation detection model.

[0085] For more detailed processing of steps S25 and S26, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0086] As can be seen, the hand image data in this embodiment is real motion information acquired using a 3D scanner or RGB-D camera. After the model outputs a six-dimensional rotation vector, Gram-Schmidt orthogonalization is used to transform the six-dimensional rotation vector to obtain the corresponding second rotation axis angle. Then, a loss function is constructed based on the first and second rotation axis angles obtained using inverse dynamics. Furthermore, in this embodiment, the developers need to pre-define the initial posture of a standard or rigid hand, which is equivalent to providing an initial value for reference. The rotation axis angles of each subsequent joint are determined accordingly based on this initial posture.

[0087] See Figure 4 As shown in the figure, this application discloses a gesture recognition method, which includes:

[0088] Step S31: Obtain the target hand image to be identified, and process the target hand image using the joint rotation detection model trained by the aforementioned disclosed joint rotation detection model training method to obtain the set of rotation axis angles of the hand joints.

[0089] In this embodiment, the target hand image to be identified is first acquired, i.e., the hand ROI (region of interest). Then, the previously trained joint rotation detection model is used to process it to obtain the set of rotation axis angles for the hand joints. Specifically, the set of rotation axis angles for all joints above the wrist is obtained. It is important to note that the positions represented by the obtained joint rotation axis angles are all based on the rotation of their respective parent nodes. Therefore, all hand joint rotations output by this model can be considered as rotations with their parent joints as the origin. Furthermore, the pose transformations of all joints can be seen as extensions of the root node rotation. The correspondence between the set Q of local rotation axis angles detected for all joints and the set F of rotation representations in the world coordinate system corresponding to that joint is as follows:

[0090]

[0091] Where, q n For each joint, the corresponding local rotation, f n For each joint, the rotation representation in the world coordinate system is defined, f0 is the rotation representation of joint 0 in the world coordinate system, and q0 is the local rotation axis angle of joint 0.

[0092] Step S32: Input the set of rotation axis angles into a preset gesture classification model to obtain gesture recognition results.

[0093] In this embodiment, the set of rotation axis angles output by the joint rotation detection model is input into a preset gesture classification model to obtain gesture recognition results. The gesture classification model can use common deep learning classification models, such as perceptron networks, multi-class SVMs, etc., or it can use template matching methods to classify gestures based on pre-set gesture pose constraints.

[0094] As can be seen, this application embodiment acquires an image of the target hand to be identified, and processes the target hand image using a joint rotation detection model trained by the aforementioned disclosed joint rotation detection model training method to obtain a set of rotation axis angles of the hand joints; the set of rotation axis angles is then input into a preset gesture classification model to obtain gesture recognition results. Thus, this application embodiment considers the rotation problem of hand joints during gesture recognition. Therefore, after acquiring the target hand image, it first processes it using a joint rotation detection model to obtain a set of rotation axis angles, and then inputs it into the gesture classification model for gesture recognition. That is, by emphasizing the rotation between various joints of the hand during gesture recognition, it avoids the recognition errors caused by changes in target size in gesture recognition based on pure image recognition. Furthermore, it is more stable than methods that directly classify based on keypoint detection, because the bone lengths of different hands are different, and the same gesture is prone to model convergence difficulties and gesture ambiguity during testing due to differences in spatial point information. Furthermore, this embodiment directly outputs the rotation axis angle through a trained joint rotation detection model, avoiding the problem of increased calculation difficulty and time consumption caused by the increase in the number of joints in existing inverse dynamics processing. In this way, the method of this application can improve gesture recognition accuracy and reduce processing time.

[0095] See Figure 5 As shown in the figure, this application discloses a joint rotation detection model training device, which includes:

[0096] Image data processing module 11 is used to acquire human hand image data, extract key points from the human hand image data to obtain first hand key points, and process the human hand image data based on the first hand key points using inverse dynamics method to obtain the first rotation axis angle of each joint of the human hand.

[0097] The rotation axis angle acquisition module 12 is used to input the human hand image data into the initial joint point rotation detection model to output a six-dimensional rotation vector, and to acquire the second rotation axis angle corresponding to the six-dimensional rotation vector;

[0098] The hand key point determination module 13 is used to determine the rotation matrix using the second rotation axis angle, and to determine the second hand key point based on the first hand key point and the rotation matrix;

[0099] The loss function construction module 14 is used to construct a first loss function based on the first rotation axis angle and the second rotation axis angle, and to construct a second loss function based on the first hand key point and the second hand key point;

[0100] The model training module 15 is used to train the initial joint rotation detection model using the first loss function and the second loss function to obtain the trained joint rotation detection model.

[0101] As can be seen, this application acquires human hand image data, extracts key points from the human hand image data to obtain first hand key points, and processes the human hand image data using inverse dynamics based on the first hand key points to obtain first rotation axis angles of each joint of the human hand; inputs the human hand image data into an initial joint rotation detection model to output a six-dimensional rotation vector, and obtains a second rotation axis angle corresponding to the six-dimensional rotation vector; uses the second rotation axis angle to determine a rotation matrix, and determines second hand key points based on the first hand key points and the rotation matrix; constructs a first loss function based on the first rotation axis angle and the second rotation axis angle, and constructs a second loss function based on the first hand key points and the second hand key points; trains the initial joint rotation detection model using the first loss function and the second loss function to obtain a trained joint rotation detection model. Therefore, this application first extracts key points from the acquired hand image data to obtain first hand key points. Based on the first hand key points, the first rotation axis angle of each joint of the hand is determined using inverse dynamics. The hand image data is input into an initial joint rotation detection model to output a six-dimensional rotation vector, which is then transformed into a corresponding second rotation axis angle. The rotation matrix determined by the second rotation axis angle and the first hand key points are then used to obtain the second hand key points. Corresponding loss functions are then constructed based on the first and second hand key points, as well as the first and second rotation axis angles, respectively. The initial joint rotation detection model is then trained using the overall loss function to obtain a trained joint rotation detection model. After extracting key points from the acquired hand image data, the above scheme further requires obtaining the rotation axis angle of the hand joints using inverse dynamics. In other words, by emphasizing the rotation between each joint point, this application avoids the problems of direct key point detection in subsequent gesture recognition and the low recognition accuracy caused by changes in hand size in pure image recognition. In addition, by training the model by fusing the loss function of the joint rotation axis angle and the loss function of the hand key points, the model constraints can be increased, thereby accelerating the convergence of the model and improving the stability and performance of the model.

[0102] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the joint rotation detection model training method performed by the electronic device disclosed in any of the foregoing embodiments.

[0103] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0104] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0105] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0106] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the joint rotation detection model training method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0107] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the method steps performed during the training process of the joint rotation detection model disclosed in any of the foregoing embodiments.

[0108] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0109] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0111] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0112] The above provides a detailed description of the model training method, gesture recognition method, device, equipment, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for training a joint rotation detection model, characterized in that, include: The process involves acquiring human hand image data, extracting key points from the human hand image data to obtain first hand key points, and processing the human hand image data based on the first hand key points using inverse dynamics to obtain the first rotation axis angle of each joint of the human hand. The hand image data is input into the initial joint rotation detection model to output a six-dimensional rotation vector, and the second rotation axis angle corresponding to the six-dimensional rotation vector is obtained. The rotation matrix is ​​determined using the second rotation axis angle, and the second hand key point is determined based on the first hand key point and the rotation matrix; The method for determining the key points of the second hand is as follows: ,in, Let be the spatial coordinates of the i-th key point in the second hand key points. and Let R be the spatial coordinates of the i-th and (i-1)-th key points in the first hand key points, and R be the rotation matrix; A first loss function is constructed based on the first rotation axis angle and the second rotation axis angle, and a second loss function is constructed based on the first hand key point and the second hand key point; The initial joint rotation detection model is trained using the first loss function and the second loss function to obtain the trained joint rotation detection model. The first loss function is: ； Where N is the total number of joints in the human hand. and These are the first rotation axis angle and the second rotation axis angle corresponding to the i-th joint point, respectively; The second loss function is: ； Where M represents the total number of key points on the hand. and These are the i-th first hand key point and the i-th second hand key point, respectively; The process of processing the human hand image data based on the first hand key points using the inverse dynamics method to obtain the first rotation axis angle of each joint of the human hand also includes: Predetermine the initial posture of the standard human hand; Based on the initial posture, a first rotation axis angle corresponding to the human hand image data is determined.

2. The joint rotation detection model training method according to claim 1, characterized in that, The acquisition of human hand image data includes: Acquire human hand image data using a 3D scanner or RGB-D camera.

3. The joint rotation detection model training method according to claim 1, characterized in that, The step of extracting key points from the human hand image data to obtain the first hand key points includes: Extract the first spatial coordinates of the hand joints and the second spatial coordinates of the fingertips from the hand image data; The first key hand point is obtained based on the first spatial coordinate and the second spatial coordinate.

4. The joint rotation detection model training method according to claim 1, characterized in that, The step of obtaining the second rotation axis angle corresponding to the six-dimensional rotation vector includes: The six-dimensional rotation vector is subjected to Gram-Schmidt orthogonalization to obtain the corresponding second rotation axis angle.

5. The joint rotation detection model training method according to claim 1, characterized in that, The step of determining the rotation matrix using the second rotation axis angle includes: The rotation matrix is ​​determined based on the rotation axis and rotation angle in the second rotation axis angle, and using the Rodriguez rotation formula.

6. A gesture recognition method, characterized in that, include: A target hand image to be identified is obtained, and the target hand image is processed using the joint rotation detection model trained by the joint rotation detection model training method as described in any one of claims 1 to 5, so as to obtain the set of rotation axis angles of the hand joints. The set of rotation axis angles is input into a preset gesture classification model to obtain gesture recognition results.

7. A joint rotation detection model training device, characterized in that, include: The image data processing module is used to acquire human hand image data, extract key points from the human hand image data to obtain first hand key points, and process the human hand image data based on the first hand key points using the inverse dynamics method to obtain the first rotation axis angle of each joint of the human hand. The rotation axis angle acquisition module is used to input the human hand image data into the initial joint point rotation detection model to output a six-dimensional rotation vector, and to acquire the second rotation axis angle corresponding to the six-dimensional rotation vector; The hand key point determination module is used to determine the rotation matrix using the second rotation axis angle, and to determine the second hand key point based on the first hand key point and the rotation matrix; The method for determining the key points of the second hand is as follows: ,in, Let be the spatial coordinates of the i-th key point in the second hand key points. and Let R be the spatial coordinates of the i-th and (i-1)-th key points in the first hand key points, and R be the rotation matrix; The loss function construction module is used to construct a first loss function based on the first rotation axis angle and the second rotation axis angle, and to construct a second loss function based on the first hand key point and the second hand key point; The model training module is used to train the initial joint rotation detection model using the first loss function and the second loss function to obtain the trained joint rotation detection model. The first loss function is: ； Where N is the total number of joints in the human hand. and These are the first rotation axis angle and the second rotation axis angle corresponding to the i-th joint point, respectively; The second loss function is: ； Where M represents the total number of key points on the hand. and These are the i-th first hand key point and the i-th second hand key point, respectively; The image data processing module is specifically used to predefine the initial posture of a standard human hand; and to determine the first rotation axis angle corresponding to the human hand image data based on the initial posture.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the joint rotation detection model training method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the joint rotation detection model training method as described in any one of claims 1 to 5.

Images