Gesture recognition virtual disassembly method based on Kinect

Abstract

The application discloses a gesture recognition virtual disassembly method based on Kinect, which comprises the following steps: collecting gesture data by using a Kinect camera; pre-training a constructed prediction model to obtain the completion degree and confidence of each gesture; inputting current gesture data into the pre-trained prediction model to obtain the confidence and completion degree of the current gesture and judging the type of the action; converting the human skeleton data in the current gesture data into a target vector in a skeletal coordinate system; taking the target vector ray as an interactive ray to perform ray collision detection to obtain the length of the ray after the ray collision; converting the skeletal coordinates into coordinates in a virtual space, and realizing synchronous interaction with a virtual object according to the type of the action and the length of the ray. The application simulates manual disassembly in a virtual reality environment, realizes control of virtual dragging, rotation and other interactions, can effectively improve the interactive experience in the virtual disassembly process, and greatly improves the efficiency of the mechanical manufacturing training process.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, and more particularly to a virtual disassembly and assembly method based on Kinect gesture recognition. Background Technology

[0002] Virtual assembly and disassembly technology is often used in practical training in the field of mechanical manufacturing, and it has the advantages of safety, economy and efficiency.

[0003] The existing virtual disassembly and assembly systems have two problems: ① They only use 2D monitors and interactive tools, which does not provide a good training effect for students; ② For disassembly and assembly courses, manual disassembly and assembly is the most realistic, but most existing disassembly and assembly systems use other interactive tools as a medium, such as mouse and keyboard input. Summary of the Invention

[0004] In view of this, in order to make virtual disassembly and assembly more realistic, this invention proposes a Kinect-based gesture recognition virtual disassembly and assembly method, including the following steps:

[0005] S1. Use the Kinect camera to collect gesture data, including image RGB data, image depth data, and human skeletal data, to obtain a gesture dataset;

[0006] S2. Construct a prediction model based on the gesture dataset, and pre-train the prediction model to obtain the completion degree and confidence degree of each gesture;

[0007] S3. Use the Kinect camera to capture the current gesture data, input it into the pre-trained prediction model in S2, obtain the confidence and completion of the current captured gesture, and determine the type of action;

[0008] S4. Establish a skeletal coordinate system, convert the human skeletal data in the current gesture data in S3 into a target vector, and optimize the target vector.

[0009] S5. Use the target vector ray as an interaction ray in Unity to perform ray collision detection, obtain the pose information of the target object and the coordinates of the collision point, and further obtain the length of the ray after the ray collision.

[0010] S6. Convert the skeletal coordinate system to the global coordinate system of the virtual space, and based on the action type determined in S3 and the pose information of the target object, the coordinates of the collision point, and the length of the ray obtained in S5, realize synchronous interaction with the virtual object.

[0011] Furthermore, step S2 specifically includes:

[0012] S21. Pre-train the gesture dataset obtained in S1 using KinectSDK;

[0013] S22. Based on the pre-trained gesture data, use the random forest algorithm to build a prediction model and obtain the completion and confidence of each gesture.

[0014] S23. Save the constructed prediction model as a .gbd format file, save the confidence and completion of each gesture, and generate its gesture recognition histogram.

[0015] Furthermore, step S3 specifically includes:

[0016] S31. Input the current gesture data collected by the Kinect camera into the prediction model stored in step S23;

[0017] S32. Calculate the confidence and completion of the current gesture;

[0018] S33. Compare the confidence and completion of the current gesture with the completion and confidence output in step S2 by a threshold to determine the type of action.

[0019] Furthermore, step S4 specifically includes:

[0020] S41. Obtain human skeletal data from the current gesture data in S3, including: shoulder data, elbow data, wrist data, and palm data;

[0021] S42. Establish a skeletal coordinate system, obtain the coordinates of the elbow and wrist joints, E and Q respectively, and calculate... vector;

[0022] S43. Optimize the target vector using the quadratic average moving method.

[0023] Furthermore, the mathematical expression for the quadratic average moving average method is:

[0024]

[0025] Where M is the window size for data movement, t represents the sampling time, and each sampling period is 90 seconds, f t To maintain the same elbow and wrist vector at sampling time t, f i To find the summation formula for f t The expression for t is given by i, which is the expression for t under the summation formula.

[0026] Furthermore, the collision detection object of the interactive ray is the collision bounding box of the virtual object.

[0027] Furthermore, step S6, which converts the skeletal coordinate system into the global coordinate system of the virtual space, is as follows:

[0028] The position of object P before transformation in the global coordinate system O of virtual space is: The relative rotation angle is The transformed position is The relative rotation angle is The position of wrist joint Q before transformation in global coordinate system O is: The relative rotation angle is The transformed position is The relative rotation angle is

[0029] Before the coordinate transformation, the position of contact point e in the global coordinate system is:

[0030]

[0031] Where f is a vector (0, 0, 1), and len is the length of the ray after the ray collision;

[0032] If object P does not rotate:

[0033]

[0034] in, Let e ​​be the vector from the contact point e to the object P. -1 This indicates finding the inverse;

[0035] Rotation of object P relative to Q for

[0036]

[0037] After coordinate transformation, the position of contact point e in global coordinates is:

[0038]

[0039] The rotation angle of object P after the transformation is

[0040]

[0041] The position of object P after the transformation is

[0042]

[0043] in, This is the straight-line distance from the wrist joint to the point of impact after the ray collision.

[0044] The beneficial effects of the technical solution provided by this invention are:

[0045] Compared with existing technologies, the gesture recognition interaction method of the present invention can simulate manual disassembly and assembly in a virtual reality environment. It realizes the control of virtual dragging, rotation and other interactions through grasping algorithms, and the difference from the disassembly and assembly mode in the real environment is small. Compared with the existing virtual disassembly and assembly system that uses mouse and keyboard input, the method implemented by the present invention can effectively improve the interactive experience in the virtual disassembly and assembly process and greatly improve the efficiency of the mechanical manufacturing training process. Attached Figure Description

[0046] Figure 1 This is a flowchart of a Kinect-based gesture recognition virtual disassembly and assembly method according to an embodiment of the present invention;

[0047] Figure 2 These are gesture images captured by Kinect in an embodiment of the present invention;

[0048] Figure 3 This is a gesture completion definition diagram automatically generated after the gesture data is imported into Kinect according to an embodiment of the present invention;

[0049] Figure 4 This is a diagram of the gesture action range according to an embodiment of the present invention;

[0050] Figure 5 This is a histogram of gesture recognition according to an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of coordinate transformation according to an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0053] refer to Figure 1 , Figure 1 This is a flowchart of a Kinect-based gesture recognition virtual disassembly / assembly method according to an embodiment of the present invention. The embodiment of the present invention provides a Kinect-based gesture recognition virtual disassembly / assembly method, including the following steps:

[0054] S1. Use the Kinect camera to collect gesture data, including image RGB data, image depth data, and human skeletal data, to obtain a gesture dataset.

[0055] In this embodiment, the gestures include waving in four directions (up, down, left, and right) and rotating the wrist joint.

[0056] S2. Construct a prediction model based on the gesture dataset, and pre-train the prediction model to obtain the completion degree and confidence degree of each gesture.

[0057] In this embodiment, the constructed prediction model sets a preset for each type of gesture to provide a detection benchmark for subsequent real-time gesture judgment. The confidence and completion of each gesture are calculated by dividing the collected gesture data into segments, showing the proportion of effective actions in the entire segment; the completion is calculated using the start and end times of the effective actions. The required data is input into KinectSDK to calculate the confidence and completion.

[0058] S21. Pre-train the gesture dataset obtained in S1 using the built-in interface of KinectSDK;

[0059] S22. Based on the pre-trained gesture data, use the random forest algorithm to build a prediction model and obtain the completion and confidence of each gesture.

[0060] S23. Save the constructed prediction model as a .gbd format file, save the confidence and completion scores for each gesture, and generate its gesture recognition histogram. The gesture recognition histogram is a visualization of the confidence and completion scores exported by KinectSDK after the above methods, which is convenient for observation during the development process.

[0061] refer to Figure 2 , Figure 3 and Figure 4 , Figure 2 These are gesture images captured by Kinect in an embodiment of the present invention. Figure 3 This is a gesture completion definition diagram automatically generated after the gesture data is imported into Kinect according to an embodiment of the present invention. Figure 4 This is a gesture action range diagram according to an embodiment of the present invention.

[0062] refer to Figure 5 , Figure 5 This is a histogram of gesture recognition in an embodiment of the present invention. The eight histograms correspond to the completion and confidence scores of the four gestures of waving hands up, down, left, and right. The specific data is shown in the table below. The confidence and completion scores of the four gestures are obtained by repeatedly collecting a single gesture with Kinect and averaging the confidence scores of the prediction models multiple times.

[0063] Confidence Completeness Right hand wave 0.65 0.75 Left hand wave 0.1 0.7 wave 0.85 0.45 wave down 0.75 0.75

[0064] S3. Use the Kinect camera to capture the current gesture data and input it into the pre-trained prediction model in S2 to obtain the confidence and completion of the captured gesture, and determine the type of action. Determining the action type is crucial for implementing different virtual disassembly / assembly interaction methods based on different gesture types.

[0065] S31. Input the current gesture data collected by the Kinect camera into the prediction model stored in step S23;

[0066] S32. Calculate the confidence and completion of the current gesture;

[0067] S33. Compare the confidence and completion of the current gesture with the completion and confidence output in step S2 by a threshold to determine the type of action.

[0068] S4. Establish a skeletal coordinate system, convert the human skeletal data in the current gesture data in S3 into a target vector, and optimize the target vector.

[0069] S41. Obtain human skeletal data from the current gesture data in S3, including: shoulder data, elbow data, wrist data, and palm data;

[0070] S42. Establish a skeletal coordinate system, obtain the coordinates of the elbow and wrist joints, E and Q respectively, and calculate... vector;

[0071] S43. Optimize the target vector using the quadratic average moving method.

[0072] The mathematical expression for the quadratic average moving average method is:

[0073]

[0074] Where M is the window size for data movement, t represents the sampling time, and each sampling period is 90 seconds, f t To maintain the same elbow and wrist vector at sampling time t, f i To find the summation formula for f t The expression for t is given by i, which is the expression for t under the summation formula.

[0075] S5. Perform ray collision detection using the target vector ray as an interactive ray in Unity to obtain the pose information of the target object and the coordinates of the collision point, and further obtain the length of the ray after the collision. In this embodiment, the collision detection object of the interactive ray is the collision bounding box of the virtual object.

[0076] S6. Convert the skeletal coordinate system to the global coordinate system of the virtual space. The purpose is to realize the synchronous movement of the coordinates of the virtual object and the skeletal coordinates. Based on the action type determined in S3 and the pose information of the target object, the coordinates of the collision point, and the length of the ray obtained in S5, the gesture action judgment and the following coordinate transformation are used to realize synchronous interaction with the virtual object.

[0077] refer to Figure 6 , Figure 6 This is a schematic diagram of coordinate transformation according to an embodiment of the present invention. The process of converting the skeletal coordinate system to the global coordinate system of the virtual space is as follows:

[0078] The position of object P before transformation in the global coordinate system O of virtual space is: The relative rotation angle is The transformed position is The relative rotation angle is The position of wrist joint Q before transformation in global coordinate system O is: The relative rotation angle is The transformed position is The relative rotation angle is

[0079] Before the coordinate transformation, the position of contact point e in the global coordinate system is:

[0080]

[0081] Where f is a vector (0, 0, 1), and len is the length of the ray after the ray collision;

[0082] If object P does not rotate:

[0083]

[0084] in, Let e ​​be the vector from the contact point e to the object P, and ()-1 indicates the inverse;

[0085] Rotation of object P relative to Q for

[0086]

[0087] After coordinate transformation, the position of contact point e in global coordinates is:

[0088]

[0089] The rotation angle of object P after the transformation is

[0090]

[0091] The position of object P after the transformation is

[0092]

[0093] in, This is the straight-line distance from the wrist joint to the point of impact after the ray collision.

[0094] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A virtual disassembly / assembly method based on Kinect gesture recognition, characterized in that, Includes the following steps: S1. Use the Kinect camera to collect gesture data, including image RGB data, image depth data, and human skeletal data, to obtain a gesture dataset; S2. Construct a prediction model based on the gesture dataset, and pre-train the prediction model to obtain the completion degree and confidence degree of each gesture; S3. Use the Kinect camera to capture the current gesture data, input it into the pre-trained prediction model in S2, obtain the confidence and completion of the current captured gesture, and determine the type of action; S4. Establish a skeletal coordinate system, convert the human skeletal data in the current gesture data in S3 into a target vector, and optimize the target vector. S5. Use the target vector ray as an interaction ray in Unity to perform ray collision detection, obtain the pose information of the target object and the coordinates of the collision point, and further obtain the length of the ray after the ray collision. S6. Convert the skeletal coordinate system to the global coordinate system of the virtual space, and based on the action type determined in S3 and the pose information of the target object, the coordinates of the collision point, and the length of the ray obtained in S5, realize synchronous interaction with the virtual object.

2. The Kinect-based gesture recognition virtual disassembly / assembly method according to claim 1, characterized in that, Step S2 is as follows: S21. Pre-train the gesture dataset obtained in S1 using KinectSDK; S22. Based on the pre-trained gesture data, use the random forest algorithm to build a prediction model and obtain the completion and confidence of each gesture. S23. Save the constructed prediction model as a .gbd format file, save the confidence and completion of each gesture, and generate its gesture recognition histogram.

3. The Kinect-based gesture recognition virtual disassembly / assembly method according to claim 2, characterized in that, Step S3 is as follows: S31. Input the current gesture data collected by the Kinect camera into the prediction model stored in step S23; S32. Calculate the confidence and completion of the current gesture; S33. Compare the confidence and completion of the current gesture with the completion and confidence output in step S2 by a threshold to determine the type of action.

4. The Kinect-based gesture recognition virtual disassembly / assembly method according to claim 3, characterized in that, Step S4 is as follows: S41. Obtain human skeletal data from the current gesture data in S3, including: shoulder data, elbow data, wrist data, and palm data; S42. Establish a skeletal coordinate system, obtain the coordinates of the elbow and wrist joints, E and Q respectively, and calculate... vector; S43. Optimize the target vector using the quadratic average moving method.

5. The Kinect-based gesture recognition virtual disassembly / assembly method according to claim 4, characterized in that, The mathematical expression for the quadratic average moving average method is: Where M is the window size for data movement, t represents the sampling time, and each sampling period is 90 seconds, f t To maintain the same elbow and wrist vector at sampling time t, f i To find the summation formula for f t The expression for t is given by i, which is the expression for t under the summation formula.

6. The Kinect-based gesture recognition virtual disassembly / assembly method according to claim 1, characterized in that, The collision detection object of interactive raycasting is the bounding box of the virtual object.

7. The Kinect-based gesture recognition virtual disassembly / assembly method according to claim 4, characterized in that, Step S6, which converts the skeletal coordinate system to the global coordinate system in the virtual space, is as follows: The position of object P before transformation in the global coordinate system O of virtual space is: The relative rotation angle is The transformed position is The relative rotation angle is The position of wrist joint Q before transformation in global coordinate system O is: The relative rotation angle is The transformed position is The relative rotation angle is Before the coordinate transformation, the position of contact point e in the global coordinate system is: Where f is the vector (0, 0, 1), and len is the length of the ray after the ray collision; if object P has not rotated: in, Let e ​​be the vector from the contact point e to the object P. -1 This indicates finding the inverse; Rotation of object P relative to Q for After coordinate transformation, the position of contact point e in global coordinates is: The rotation angle of object P after the transformation is The position of object P after the transformation is in, This is the straight-line distance from the wrist joint to the point of impact after the ray collision.

Images