Character modeling and fitness action teaching method and system based on non-contact action capture

Through the combination of millimeter-wave radar and deep learning neural networks, the equipment dependence and accuracy of existing motion capture technology are solved, and high-precision contactless motion capture and fitness movement scores are achieved, which improves user experience and widespread application.

CN120563731APending Publication Date: 2025-08-29UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202510657385.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-21
Publication Date
2025-08-29

AI Technical Summary

Technical Problem

The existing motion capture technology has strong dependence, poor real-time performance, insufficient accuracy and is not convenient for widespread application, especially optical and sensor motion capture technology, which is sensitive to the environment, limited user activities and high cost.

Method used

Millimeter-wave radar and deep learning neural network are used to combine computer vision technology to obtain human posture data for contactless motion capture, reconstruct 3D human postures, use Kalman filtering and IK inverse kinematics to perform data smoothing and joint angle calculation, and combine Google image processing human posture framework for data fusion.

Benefits of technology

It realizes high-precision contactless motion capture, supports fitness movement scores and real-time feedback, improves users' freedom of activity and convenience of use, and is widely used in fields such as virtual fitness and sports rehabilitation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120563731A_ABST
    Figure CN120563731A_ABST
Patent Text Reader

Abstract

The invention discloses a figure modeling and fitness action teaching method and system based on non-contact action capture, and the method carries out the real-time monitoring of a human body posture through a millimeter wave radar, carries out the 3D human body posture reconstruction and analysis through combining with a human body posture frame of Google image processing, and achieves the high-precision action capture. The method can be used for human body modeling, body building action scoring and action guidance, action evaluation and real-time feedback in body building teaching can be effectively supported, the millimeter wave radar is adopted for human body posture capture, external markers or wearable equipment are not needed, constraint on user activities is avoided, and use convenience and comfort are improved. The 3D posture of the human body can be accurately reconstructed in combination with a deep learning neural network, and real-time processing and analysis are carried out. The method not only can realize accurate motion capture, but also can be combined with fitness motion scoring and guidance, is widely applied to the fields of virtual fitness, exercise rehabilitation and the like, and has great application potential.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of non-contact motion capture, in particular to a method and system for character modeling and fitness movement teaching based on non-contact motion capture. Background Art

[0002] With the rapid development of technologies such as virtual reality, augmented reality, and smart fitness, motion capture technology has been widely applied in various fields. Existing motion capture technologies are mainly divided into two categories: optical-based motion capture and sensor-based motion capture. Optical motion capture requires the installation of a large number of devices within the capture area and the use of special reflective markers. It is also very sensitive to lighting conditions and is easily affected by environmental changes, limiting its use in certain scenarios.

[0003] Sensor-based motion capture technology can improve capture accuracy and applicability to a certain extent. However, these sensors typically need to be worn by the user, limiting their freedom of movement. Furthermore, sensors are prone to errors over long periods of use, and the system costs are high.

[0004] In view of the shortcomings of the existing technology, a non-contact motion capture method and system are needed. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a character modeling and fitness movement teaching method and system based on non-contact motion capture. The method realizes non-contact motion capture through millimeter wave radar, deep learning neural network and computer vision, obtains human body posture data in real time through millimeter wave radar, and reconstructs human body 3D posture in combination with deep learning technology.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The character modeling method based on non-contact motion capture provided by the present invention comprises the following steps:

[0008] S1: Acquire point cloud data of the target through millimeter-wave radar; acquire visual images of the target through a wide-angle camera;

[0009] S2: Process the radar point cloud to identify human features; process the visual image, extract the human joints collected in the visual image, and use computer vision technology to identify and locate the positions of human joints from the image;

[0010] S3: Utilize deep learning to reconstruct human joints, and obtain joint fusion data by reconstructing human joints based on millimeter-wave radar and the human posture framework based on image processing;

[0011] S4: Character model construction: Obtain a 3D human body model based on joint fusion data.

[0012] Furthermore, the method further comprises the following steps:

[0013] S5: Use the obtained 3D human body model to obtain the motion matrix of different time series and analyze and process it with the standard motion matrix to obtain motion quality evaluation.

[0014] Furthermore, in step S3, the joint fusion data is obtained in the following manner:

[0015] The human body position and motion trajectory are estimated through Kalman filter tracking; the human body position and motion trajectory are smoothed using least squares smoothing; and the joint angles required to calculate the skeleton model to perform specific actions are obtained through IK inverse kinematics solution.

[0016] Furthermore, the image processing in step S3 adopts the human posture framework of Google Image Processing.

[0017] Furthermore, the human joint reconstruction is achieved by using deep learning in step S3, and the training is specifically performed in the following manner:

[0018] Dataset Construction: The denoised point cloud was processed using a grid voxelization method. The point cloud in three-dimensional space was projected onto the x-y, x-z, and y-z planes as the dataset features. Simultaneously acquired video data was input into a human pose framework based on Google Image Processing, and the output human joint points were used as dataset labels. A deep learning model was trained to reconstruct human joints by learning the mapping relationship between point cloud data and pose data (joint points).

[0019] The deep learning model is constructed as follows: Based on the point cloud data and the corresponding joint labels, the deep network architecture of ResNet-18 is used to extract features and predict joint positions;

[0020] The constructed dataset is input into the ResNet-18 network to learn the mapping relationship between the voxel projection of point cloud data and the position of human joints;

[0021] Construct the mean square error (MSE) loss function to measure the error between the joint coordinates predicted by the model and the actual coordinates;

[0022]

[0023] in, is the spatial coordinate of the i-th joint predicted by the model, y i is the real coordinate of the i-th joint, and N is the total number of joints.

[0024] Furthermore, the 3D human body model is trained in the following manner:

[0025] Use the IK numerical solution algorithm to calculate the root joint and the angle of each joint from the target position. According to the target position bone structure, the angle of each joint is calculated by the algorithm;

[0026] The skeleton structure is used to establish a character model in Blender modeling software through bone connection. The character model is a hierarchical structure consisting of multiple joints and bones. Each joint has an initial position and a rotation angle, and adjacent joints are connected by bones.

[0027] According to the physical constraints of the human skeleton, the limit range of joint rotation is set, the IK solving algorithm is applied to optimize the Euler angles of the skeleton, the rotation angle of each joint is adjusted to minimize the target error, and the LBFGS optimizer is used to minimize the error between the target position of each pair of joints and the position calculated by the model. By applying L2 regularization to the Euler angles, the overfitting problem of the model is reduced and the posture is kept smooth.

[0028] The present invention provides a method for teaching fitness movements based on character modeling using non-contact motion capture, comprising the following steps:

[0029] S51: Obtaining a standard motion matrix and obtaining a user motion matrix according to the 3D human body model;

[0030] S52: Calculate the similarity between the standard action matrix and the user action matrix;

[0031] S53: Determine whether the similarity exceeds a similarity threshold V1 based on the similarity. If it does not exceed the threshold V1, the current action is not performed, and the current action angle matrix is ​​calculated. If it exceeds the threshold V1, determine whether the similarity threshold V2 is reached. If it does not exceed the threshold V2, determine that the action is not standard, and calculate the current action angle matrix. If it exceeds the threshold V2, determine that the action is standard, and proceed to the next step S54.

[0032] S54: Determine whether there is a next frame action. If yes, import the next frame standard action angle matrix; if not, end the learning;

[0033] S55: Return to step S51 and continue the cycle.

[0034] Furthermore, it also includes fitness action scoring, which is specifically carried out according to the following steps:

[0035] S61: extracting the bone rotation angle of the standard motion to form a standard motion data matrix;

[0036] S62: extracting the skeletal rotation angle of the user's actual movement to form a user action data matrix;

[0037] S63: Use the DTW algorithm to calculate the similarity between the standard action data matrix and the user action data matrix.

[0038] Furthermore, the DTW algorithm is performed in the following manner:

[0039] The input data includes user action matrix and standard action matrix;

[0040] User action matrix: U∈R T×45 ;

[0041] Standard action matrix:

[0042] The local distance between the user action matrix and the standard action matrix is ​​calculated according to the following formula:

[0043]

[0044] Among them, w k is the joint weight;

[0045] U i,k,m For the mth Euler angle component of the user's i-th frame and k-th joint, the dynamic programming recursive formula is:

[0046]

[0047] Normalized similarity calculation:

[0048]

[0049] The user's action scoring rules can be obtained based on the similarity calculation of each frame.

[0050] Furthermore, the action scoring rule adopts single-frame scoring or overall scoring;

[0051] The single-frame scoring formula is:

[0052]

[0053] The overall score is the weighted average of all frame scores, with the key frame weight doubled.

[0054] The present invention provides a fitness movement teaching system for character modeling based on non-contact motion capture, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor implements the above method when executing the program.

[0055] The beneficial effects of the present invention are:

[0056] The present invention provides a method and system for character modeling and fitness movement teaching based on non-contact motion capture. The method monitors human posture in real time through millimeter-wave radar, and reconstructs and analyzes 3D human posture in combination with the human posture framework of Google image processing, thereby achieving high-precision motion capture. It can be used for human body modeling, fitness movement scoring and movement guidance, and can effectively support movement evaluation and real-time feedback in fitness teaching. Since millimeter-wave radar is used for human posture capture, there is no need to rely on external markers or wearable devices, which avoids constraints on user activities and improves the convenience and comfort of use. Combined with a deep learning neural network, it can accurately reconstruct the 3D posture of the human body and perform real-time processing and analysis. It can not only achieve accurate motion capture, but also be combined with fitness movement scoring and guidance. It is widely used in virtual fitness, sports rehabilitation and other fields, and has great application potential.

[0057] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration.

[0059] Figure 1 This is the system architecture diagram.

[0060] Figure 2 Flowchart for radar point cloud training.

[0061] Figure 3 For non-contact motion capture and character modeling.

[0062] Figure 4 The human skeleton after fusion.

[0063] Figure 5 Flowchart of the fitness teaching algorithm.

[0064] Figure 6 Schematic diagram of user fitness interaction. DETAILED DESCRIPTION

[0065] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0066] Example 1

[0067] like Figure 1As shown, the present embodiment provides a method for character modeling and fitness movement teaching based on non-contact motion capture. This method uses millimeter-wave radar and Google image processing to construct a human posture framework, solving the problems of existing motion capture technology such as strong equipment dependence, poor real-time performance, insufficient accuracy, and inconvenience for widespread application.

[0068] The character modeling method based on non-contact motion capture provided in this embodiment includes the following steps:

[0069] S1: At the hardware perception layer: millimeter-wave radar is used to obtain point cloud data of the target to be measured; a wide-angle camera is used to obtain a visual image of the target to be measured; this embodiment requires camera calibration before using the wide-angle camera; thereby achieving accurate capture of human movements. Millimeter-wave radar is good at penetrating fog, smoke, and some obstacles to detect distant objects; the wide-angle camera provides high-resolution visual images, suitable for capturing details and movements.

[0070] S2: In the data processing layer: radar point cloud data acquired by radar equipment is used to describe the three-dimensional structure of the surrounding environment. Processing this data can identify human features, especially the moving parts; visual images are processed to extract human joints captured in the visual images. Computer vision technology is used to identify and locate the positions of human joints in the images, making them useful for subsequent analysis and action recognition;

[0071] S3: In the modeling fusion layer: deep learning is used to reconstruct human joints. Human joint fusion data is obtained by reconstructing human joints based on millimeter-wave radar and the human posture framework based on Google image processing.

[0072] The human body position and motion trajectory are estimated through Kalman filter tracking; the human body position and motion trajectory are smoothed using least squares smoothing (MLS) to reduce the influence of noise; the joint angles required to calculate the skeleton model to perform specific actions are obtained through IK inverse kinematics solution;

[0073] S4: Character model construction: A 3D human body model is obtained based on the joint fusion data; this embodiment integrates multiple sensor data (camera visual images and radar data) into a unified and accurate 3D human body model.

[0074] S5: At the application layer: the obtained 3D human body model is used to obtain the motion matrices of different time series and analyze and process them with the standard motion matrix to obtain motion quality assessment; the application layer of this embodiment is the final user interface of the entire system, integrating the underlying algorithms and technologies to provide users with intuitive functions such as watching motion replays, checking scores, receiving correction suggestions, etc.

[0075] The motion quality assessment provided in this embodiment uses the Dynamic Time Warping (DTW) method. DTW is a method for comparing two time series, even if they have different speeds. In motion capture, similar motion patterns are identified by aligning and comparing different motion templates.

[0076] In this embodiment, human motion data acquisition uses a millimeter-wave radar sensor on a 2.1m-high tripod to perform real-time human motion data acquisition, capturing point cloud information of the human body in space. The collected data is preprocessed using a polar coordinate conversion algorithm to convert the point cloud coordinates in the polar coordinate system to rectangular coordinates in the rectangular coordinate system. Multi-frame dynamic point compensation is performed on the point cloud coordinates in the rectangular coordinate system to reduce the instability of dynamic point cloud data and improve data density and accuracy. The point cloud is then denoised using the DBSCAN algorithm, a density-based clustering algorithm that can effectively identify and remove sparse noise points, thereby obtaining cleaner point cloud data.

[0077] The dataset constructed in this embodiment uses a grid voxelization method to process the denoised point cloud. The point cloud in three-dimensional space is projected onto the x-y, x-z, and y-z planes as the dataset features. Simultaneously acquired video data is input into a human pose framework based on Google Image Processing, and the output human joint points are used as dataset labels. By learning the mapping relationship between point cloud data and pose data (joint points), a deep learning model is trained to reconstruct human joints.

[0078] The deep learning model in this embodiment is constructed based on point cloud data and corresponding joint labels, using the ResNet-18 deep network architecture to extract features and predict joint positions. ResNet-18 is an 18-layer residual network that introduces residual connections to avoid the vanishing gradient problem, allowing the model to be trained at deeper levels and maintain high performance.

[0079] The constructed dataset is input into the ResNet-18 network to learn the mapping relationship between the voxel projection of the point cloud and the position of the human joints;

[0080] The mean square error (MSE) loss function is constructed to measure the error between the joint coordinates predicted by the model and the actual coordinates.

[0081]

[0082] Where i represents the index of the joint, is the spatial coordinate of joint i predicted by the model, y i is the real coordinate of joint i, N is the total number of joints, and in this embodiment, N=17.

[0083] The calibration of the wide-angle camera in this embodiment uses a checkerboard calibration method to identify checkerboard corners in the image to achieve camera calibration, obtain the camera's internal and external parameters, and compensate for lens distortion, thereby improving the geometric accuracy of the image; the checkerboard calibration method uses a standard checkerboard pattern as a calibration plate. The checkerboard is usually composed of black and white squares. The side length of each square is used as a reference size for calibration, and the camera is used to shoot calibration plate images at multiple angles and positions. A computer vision algorithm is used to extract the intersection (corner point) position of each square of the checkerboard in each image. Combined with the corner point coordinates in the image, a calibration algorithm is used to calculate the camera's internal and external parameters, and the image is subjected to distortion correction using the camera's internal and external parameters obtained by the calibration algorithm.

[0084] The millimeter-wave radar reconstruction of human joints in this embodiment is fused with the human posture framework recognition joints based on Google image processing using a Kalman filter, which can perform real-time estimation of dynamic systems and is suitable for smoothing and predicting information such as position and speed; the Kalman filter fuses the joint coordinates output by the radar point cloud and the joint coordinates collected by the camera as observation quantities.

[0085] Assuming that the position of each joint can be modeled by a dynamic system, the Kalman filter calculates the optimal estimate based on the predicted value at the previous moment and the current observation value (joint position from two sensors); through the Kalman filter, the system can not only obtain the joint coordinates based on the observation data of the current frame, but also smooth it in combination with the data of the previous frame to obtain a more stable and accurate joint position output; in order to reduce the twitching of the bones between each frame after fusion, the human posture data is smoothed using the moving least squares (MLS) algorithm to obtain the final fused human joints.

[0086] The character model in this embodiment is constructed using the IK numerical solution algorithm to calculate the root joint and the angle of each joint from the target position. Based on the target position bone structure, the angle of each joint is calculated by the algorithm;

[0087] The skeleton structure is used to establish a character model through bone connection in the Blender modeling software. The character model consists of a hierarchical structure consisting of multiple joints and bones. Each joint has an initial position and rotation angle, and adjacent joints are connected by bones. The skeleton structure is usually a tree structure composed of multiple joints, and the root joint corresponds to the root of the skeleton; according to the physical constraints of the human skeleton (such as the minimum and maximum values ​​of the joint angles), the limit range of joint rotation is set, and the IK solution algorithm is applied to optimize the Euler angles of the bones. The rotation angle of each joint is adjusted to minimize the target error. The LBFGS optimizer is used to minimize the error between the target position of each pair of joints and the calculated position of the model. By applying L2 regularization to the Euler angles, the overfitting problem of the model is reduced and the posture is kept smooth.

[0088] The following describes in detail the specific implementation process of the non-contact motion capture method. The specific process is as follows:

[0089] This embodiment uses deep learning to implement the millimeter wave radar human joint reconstruction process as follows:

[0090] The S31 uses a 77GHz millimeter-wave radar from Skerry Technologies, mounted on a 2.1m tripod. It covers a 120° horizontal field of view and a 30° vertical field of view, capturing real-time point cloud data of human motion. It simultaneously records video at 30fps using a 160° wide-angle camera, and uses a Google Image Processing-based human pose framework to generate joint point labels.

[0091] S32: Data processing. The human body posture data collected by the camera is the 3D spatial coordinate points of the human body joints. In order to reduce the twitching of the human body joints between each frame, the human body posture data is first smoothed using the moving least squares algorithm. The formula is as follows:

[0092]

[0093] in, is the weight function, P i (x) is the interpolation polynomial; d(x,p i ) represents the target point x and the neighboring data points p i The Euclidean distance between them is used to measure the similarity or association strength between the two; f(x) represents the coordinate value of the target point x after smoothing;

[0094] The human body posture data of each frame consists of the three-dimensional coordinates of each joint, denoted as P i (x), where i is the joint number. For each data point x, calculate its distance d(x,p i ), and then calculate the corresponding weight w according to the selected weight function i(x). For each joint’s 3D coordinate, calculate the interpolation polynomial P i (x). According to the formula The weighted interpolation results of all data points are summed to obtain the smoothed result.

[0095] For 3D point cloud data, the DBSCAN method is used for filtering. This method can evaluate the similarity between points through the Euclidean distance metric and remove noise points. A point p is a core point. MinPts is used to define the density threshold of the core point if and only if the number of points in the neighborhood of the core point is greater than or equal to MinPts; that is, there are at least MinPts points within the ε radius. For any two points p = (x p ,y p ,z p ) and q=(x q ,y q ,z q ), the Euclidean distance d(p,q) between them is calculated as:

[0096]

[0097] Define the neighborhood of a point p as: N ε (p)={q|d(p,q)≤ε}

[0098] Among them, N ε (p) is the ε-neighborhood of point p, which contains all points whose distance to point p is less than or equal to ε;

[0099] If, |N ε (p)|≥MinPts, then point p is a core point;

[0100] If the point is a core point, a new cluster is created, containing the point and all points in its neighborhood. For each core point in the neighborhood, the neighborhood is expanded until no new points can be added. A boundary point is located in the neighborhood of a core point, but is not itself a core point. If a point is neither a core point nor in the neighborhood of any core point, it is marked as a noise point. Ultimately, the noise in the 3D point cloud can be removed. Multi-frame motion compensation is performed on the 3D point cloud after the noise is removed: 5 adjacent frames of point cloud data are combined into a single frame to obtain a denser point cloud information.

[0101] S33: Dataset construction. The point cloud data is grid-voxelized, and the voxel size is divided into 0.1m×0.1m×0.1m. It is projected onto the XY, YZ, and XZ planes to generate three-channel images (resolution 256×256). The label data is the coordinates of 17 joint points output by the human posture framework based on Google Image Processing (normalized to [-1,1]).

[0102] S34: Model training process;

[0103] like Figure 2 As shown, Figure 2 This is a flowchart for radar point cloud training. The training data is the original 3D point cloud collected by millimeter-wave radar. Each point contains spatial coordinates (x, y, z) representing the sparse distribution of human joints in the environment.

[0104] Further, based on the representation of point cloud data in three-dimensional space, it is projected onto three sets of orthogonal planes: x-y, x-z, and y-z. The scene grid is discretized and the point cloud data information is stored in three sets of data matrices. At the same time, the joint information coordinates captured by the motion capture device are used as labels to complete the network input and label part of the dataset. The number of radar frames is divided into training set and validation set.

[0105] Furthermore, the spatial segmentation network uses the convolutional layer as the backbone to achieve channel information fusion, integrates SE attention and deep high-resolution representation modules, extracts spatial features from point cloud projections, and fuses the information obtained from three sets of orthogonal planes to obtain the target X;

[0106] Furthermore, the three-dimensional convolutional neural network uses ResNet18 as the network backbone, and is formed by stacking four groups of residual blocks with CA attention mechanism. It performs deep mining based on the multi-scale features obtained from channel information fusion, completes high-density feature representation, obtains feature encoding vectors, and realizes deep feature mining.

[0107] Furthermore, a multi-layer perceptron is used to process the feature coding vector described in the feature depth mining, and the joint points representing the skeleton of the human body posture in the current frame are output to achieve posture decoding.

[0108] This example constructs a ResNet-18 model for model training. The input is a three-channel voxel projection image, and the output is the three-dimensional coordinates (x, y, z) of 17 joint points. The loss function is the mean squared error (MSE), the optimizer is Adam, the initial learning rate is 0.001, the batch size is 32, and the training is 3000 epochs.

[0109] The calibration process of the wide-angle camera in this embodiment is as follows:

[0110] S11: Use a 10×7 chessboard (square length 3 cm) to capture 20 images of the calibration plate in different poses.

[0111] S12: Preliminary corner point location is performed through edge detection and Hough transform. The corner points of the checkerboard calibration plate are detected. Extracting corner points is a preprocessing step for camera calibration. Based on multiple sets of checkerboard corner point data, the camera intrinsic parameter matrix and distortion coefficients (radial distortion coefficients k1 and k2, tangential distortion coefficients p1 and p2) are calculated to establish a camera imaging model.

[0112] S13: Based on the intrinsic parameter matrix and distortion coefficient obtained by camera calibration in S12, the input image is geometrically corrected to eliminate lens distortion and perform real-time distortion correction on the video stream.

[0113] like Figure 3 As shown, Figure 3 For non-contact motion capture and character modeling, the specific process of sensor data fusion is as follows:

[0114] Collect 3D radar point cloud data and feed it into the constructed model for model training to obtain the human body posture reconstruction from the radar point cloud;

[0115] Collect real-time video streams and feed them into the human posture framework based on Google image processing to obtain the human posture reconstruction of the visual image;

[0116] The character modeling and joint angle matrix sequence are obtained through Kalman filter fusion and IK inverse kinematics solution;

[0117] This embodiment uses Kalman filtering to fuse the human joint coordinates output by the 3D radar point cloud with the human joint coordinates of the human posture framework based on Google image processing.

[0118] like Figure 4 Figure 4 As shown, Figure 4 This is a schematic diagram of the effect of the human skeleton after fusion. Kalman filtering is a recursive algorithm that can perform real-time estimation of dynamic systems and is suitable for smoothing and predicting information such as position and velocity.

[0119] The Kalman filter fuses the joint coordinates output by the radar point cloud and the joint coordinates collected by the camera as observations.

[0120] Assuming that the position of each joint can be modeled as a dynamic system, the Kalman filter calculates the optimal estimate based on the predicted value at the previous moment and the current observation value (joint position from two sensors);

[0121] Through Kalman filtering, the system can not only obtain joint coordinates based on the observation data of the current frame, but also smooth it with the data of the previous frame to obtain a more stable and accurate joint position output;

[0122] The basic formula of Kalman filtering includes two steps: prediction and update;

[0123] State prediction formula:

[0124]

[0125] Forecast covariance matrix:

[0126] Pk|k―1 =AP k―1|k―1 A T +Q

[0127] Here is the predicted value of the joint position of k at the current moment, A is the state transfer matrix, which is the linear change of the joint position and the motion model of the joint between two frames. is the estimated joint position at the previous moment, P k|k―1 is the prediction covariance matrix, Q is the process noise covariance;

[0128] Then update the data after prediction. The following three formulas are needed for the update.

[0129] Kalman gain calculation:

[0130] K k =P k|k―1 H T (HP k|k―1 H T +R) ―1

[0131] Status Update:

[0132]

[0133] Update the covariance matrix:

[0134] P k|k =(I-K k H)P k|k―1

[0135] Among them, K k represents the Kalman gain, which indicates how to combine the predicted value and the observed value. It balances the uncertainty of the prediction and the reliability of the observation; represents the updated joint position, which is the optimal estimate after fusing the human pose data reconstructed by the radar and the human pose frame data processed based on Google Image; H is the identity matrix, which represents the direct observation of the joint position; R is the observation noise covariance, which is set to 0 here; I is the identity matrix;

[0136] The IK solving and character model building process in this embodiment is as follows:

[0137] In this embodiment, after the monocular camera and millimeter-wave radar data are fused and filtered, the obtained data information is subjected to an inverse kinematics (IK) solution algorithm to calculate the joint angle from the target position.

[0138] The human skeleton is composed of multiple rigid links and joints. The relative positions and angles of the joints and links are θ1, θ2, …, θ n , let the desired position of the target end effector be ptarget , then the desired position of the end effector p end (θ1,θ2,…,θ n )=p target .

[0139] In this example, the Jacobian matrix is ​​used to solve the problem, and the formula is:

[0140] Δp end =J(θ)Δθ

[0141] but:

[0142] Δθ=J(θ) ―1 Δp end

[0143] Then you can update the angle:

[0144] θ new =θ+αΔθ

[0145] J(θ) represents the Jacobian matrix of the end effector position versus joint angle (dimension 3×n);

[0146]

[0147] Where θ represents the current joint angle set θ=(θ1,θ2,…,θ n );

[0148] Δθ represents the set of angle increments Δθ=(Δθ1, Δθ2, …, Δθ n );

[0149] θ new represents the updated joint angle θ new =θ+αΔθ;

[0150] 0<α<1 means the step size factor is used to control the iterative convergence speed;

[0151] And so on, to determine the angle of the human skeleton;

[0152] In this embodiment, the theory of motion wave theory is used to simulate the relative motion between joints and limbs in the human skeleton;

[0153] The formula is:

[0154]

[0155] Where u(x,t) is the displacement field, describing the motion of the joint or skeleton; c is the wave velocity (related to the motion characteristics of the body); is the Laplace operator, describing the spatial variation of wave propagation, and t is the time variable; it is used to track the instantaneous state of wave propagation in the skeleton system;

[0156] The L-BFGS optimizer was used with 50 iterations and L2 regularization (coefficient 0.01) was added to prevent overfitting.

[0157] According to the above method, the coordinates and angles of human joints can be derived.

[0158] In this example, a character model is created using skeletal connections in Blender modeling software. The character model is a hierarchical structure consisting of multiple joints and bones, including 15 joints (the root joint is the hip). Each joint has an initial position and rotation angle, and adjacent joints are connected by bones. The skeleton structure is generally a tree-like structure composed of multiple joints, with the root joint corresponding to the root of the skeleton.

[0159] Example 2

[0160] This embodiment also provides a non-contact motion capture fitness teaching system, including a data acquisition module, a data analysis module, a model building module, and a fitness teaching module;

[0161] The data acquisition module includes a 77GHz millimeter wave radar and a monocular wide-angle camera;

[0162] The 77GHz millimeter wave radar is used to collect three-dimensional echoes of the human body and construct a three-dimensional point cloud of the human body;

[0163] The monocular wide-angle camera is used to collect human motion video stream;

[0164] The data analysis module processes the collected millimeter wave radar point cloud and human motion video stream into human joint points;

[0165] The model building module calculates the root bone coordinates of the human body and the rotation angles of the remaining bones through the IK solution algorithm using the joint coordinates output by the data analysis module.

[0166] Example 3

[0167] like Figure 5 As shown, Figure 5 This is a flowchart of a fitness teaching algorithm. In this embodiment, the obtained 3D human body model is applied to the fitness field. The specific implementation process of the fitness teaching method is described in detail below, which specifically includes the following steps:

[0168] S51: Obtaining a standard motion matrix and obtaining a user motion matrix according to the 3D human body model;

[0169] S52: Calculate the similarity between the standard action matrix and the user action matrix;

[0170] S53: Determine whether the similarity exceeds the similarity threshold V1 based on the similarity. If it does not exceed the threshold V1, the current action is not performed, and the current action angle matrix is ​​calculated. If it exceeds the threshold V1, determine whether the similarity threshold V2 is reached. If it does not exceed the threshold V2, it is determined that the action is inaccurate, and the current action angle matrix is ​​calculated. If it exceeds the threshold V2, it is determined that the action is standard, and enter the next step S54.

[0171] S54: Determine whether there is a next frame action. If yes, import the next frame standard action angle matrix; if not, end the learning;

[0172] S55: Return to step S51 and continue the cycle.

[0173] Specifically, in this embodiment, different values ​​of V1 and V2 are set according to different learning modes.

[0174] Furthermore, mode 1 is the novice mode. If the user achieves 40% similarity, it is considered that the user is performing the current action, and 70% similarity is considered as the standard action.

[0175]

[0176] Furthermore, mode 2 is an advanced mode where if the user achieves 50% similarity, it is considered that the user is performing the current action, and if it achieves 80% similarity, the action is standard.

[0177]

[0178] Furthermore, mode three is an advanced mode where if the user achieves 50% similarity, it is considered that the user is performing the current action, and if the user achieves 85% similarity, the action is standard.

[0179]

[0180] The fitness teaching method also includes fitness movement scoring, which is specifically carried out according to the following steps:

[0181] S61: extracting the bone rotation angle of the standard motion to form a standard motion data matrix;

[0182] S62: extracting the skeletal rotation angle of the user's actual movement to form a user action data matrix;

[0183] S63: Use the DTW algorithm to calculate the similarity between the standard action data matrix and the user action data matrix and score them;

[0184] The fitness teaching method also includes fitness movement teaching, and the fitness movement teaching provides correction suggestions for the movement based on the similarity calculation score;

[0185] If the similarity is higher than the similarity threshold V1, which is the basic threshold, it is determined that the user is currently learning the standard action of the frame; otherwise, the user is not performing the current action, and the user action data matrix is ​​recalculated and S61 is repeated;

[0186] Detect whether the similarity threshold between the standard action data matrix and the user action data matrix is ​​higher than V2 as the standard threshold. If it is higher than V2, it is the current action standard; if it is not higher than V2, recalculate the user action data matrix and repeat S61;

[0187] Check whether there is a standard action data matrix for the next frame in the standard action library. If yes, read it and repeat S61. If no, the learning ends.

[0188] The fitness action scoring algorithm in this embodiment adopts the DTW dynamic time warping algorithm, and the input data includes the user action matrix and the standard action matrix;

[0189] User action matrix: 3D Euler angles of 17 joints (3 dimensions per joint);

[0190] Standard action matrix:

[0191] Calculate the local distance between the user action matrix and the standard action matrix: use the modified Euclidean distance, taking into account the joint weights (such as the shoulder and hip joint weight coefficient is 1.5, and the rest is 1.0):

[0192]

[0193] Where d(i,j) is the local distance, which represents the difference between the user's action in frame i and the standard action in frame j (the smaller the distance, the more similar they are); w k U is the joint weight (e.g., w = 1.5 for the hip joint and w = 1 for the rest). In this embodiment, higher weights are assigned to key joints (e.g., hip and shoulder) because they have a greater impact on the quality of the movement. i,k,m is the dynamic programming recursive formula, which represents the mth Euler angle value of the user's i-th frame and k-th joint; i∈[1,T] represents the i-th frame of the user's action (T is the total number of user action frames); k∈[1,17] represents the k-th joint; m∈[1,3] represents the mth dimension of the Euler angle (pitch, yaw, roll angle); S j,k,m Indicates the theoretical value of the Euler angle m of joint k at the jth frame in the standard action library; j∈[1,T′], index k, m and U i,k,m In this example, k and m are the same.

[0194] Furthermore, dynamic programming is used to gradually match the U and S sequences to solve the problem of time axis asynchrony between the user and the standard action (such as the difference in action speed);

[0195]

[0196] D(i,j) is the cumulative distance, representing the minimum cumulative distance from the user's action start frame to the i-th frame, and from the standard action start frame to the j-th frame. D(i-1,j), D(i,j-1), and D(i-1,j-1) are path backtracking terms, where D(i-1,j) indicates that the user's action is slower than the standard action (the user takes one more frame to complete the same progress), D(i,j-1) indicates that the user's action is faster than the standard action (the user takes one less frame to complete the same progress), and D(i-1,j-1) indicates that the user's action is synchronized with the standard action.

[0197] Further normalized similarity calculation is performed:

[0198]

[0199] Among them, D max is the theoretical maximum distance, V is the normalized similarity, V = 1 means a perfect match, and V = 0 means no match at all.

[0200] Further, based on the similarity calculation of each frame, the user's action scoring rules can be obtained, which can be single-frame scoring or overall scoring;

[0201] The single-frame scoring formula is:

[0202]

[0203] Among them, V1 represents the basic threshold; V2 represents the standard threshold; Score i Indicates the real-time score of the user's action in frame i i ∈[0,100]; In this embodiment, the score is based on the single frame i Perform piecewise linear mapping: Score i =0 means that the basic threshold V1 is not reached (action is not recognized); Score i ∈[50,99] indicates that the action exists but needs to be corrected; Score i =100 means the action meets the standard.

[0204] The overall score is the weighted average of all frame scores, with the key frame weight doubled.

[0205]

[0206] Among them, Score total Indicates the overall score, w i represents the weight of the i-th frame. Specifically, in this embodiment, the key frame weight (such as the squat action frame) is set to 2.0, and the transition frame is set to 1.0).

[0207] like Figure 6 As shown, Figure 6 The following is a user fitness interaction diagram. The specific process of the fitness movement teaching algorithm in this embodiment is as follows:

[0208] Compare the standard action matrix and the user action matrix using the DTW algorithm similarity;

[0209] The similarity threshold V1 = 60% is used to determine whether the user has started to perform the target action. If the similarity is lower than V1, the system considers that the user has not followed the instruction. The threshold for determining whether the action has met the standard is V2 = 85%.

[0210] When it is higher than V2, the current frame action is considered to meet the standard; read whether there is a standard action matrix for the next frame in the standard action library. If so, read it and repeat the above steps. If not, the learning ends.

[0211] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A character modeling method based on non-contact motion capture, characterized by: The following steps are involved: S1: Acquire point cloud data of the target through millimeter-wave radar; acquire visual images of the target through a wide-angle camera; S2: Process the radar point cloud to identify human features; Process the visual image, extract the human joints collected in the visual image, and use computer vision technology to identify and locate the position of the human joints from the image; S3: Utilize deep learning to reconstruct human joints, and obtain joint fusion data by reconstructing human joints based on millimeter-wave radar and the human posture framework based on image processing; S4: Character model construction: Obtain a 3D human body model based on joint fusion data.

2. The character modeling method based on non-contact motion capture according to claim 1, wherein: The following steps are also included: S5: Use the obtained 3D human body model to obtain the motion matrix of different time series and analyze and process it with the standard motion matrix to obtain motion quality evaluation.

3. The character modeling method based on non-contact motion capture according to claim 1, wherein: In step S3, the joint fusion data is obtained in the following manner: The human body position and motion trajectory are estimated through Kalman filter tracking; the human body position and motion trajectory are smoothed using least squares smoothing; and the joint angles required to calculate the skeleton model to perform specific actions are obtained through IK inverse kinematics solution.

4. The character modeling method based on non-contact motion capture according to claim 1, wherein: The step S3 of using deep learning to achieve human joint reconstruction is specifically trained in the following manner: Dataset Construction: The denoised point cloud was processed using a grid voxelization method. The point cloud in three-dimensional space was projected onto the x-y, x-z, and y-z planes as the dataset features. Simultaneously acquired video data was input into a human pose framework based on Google Image Processing, and the output human joint points were used as dataset labels. A deep learning model was trained to reconstruct human joints by learning the mapping relationship between point cloud data and pose data (joint points). The deep learning model is constructed as follows: Based on the point cloud data and the corresponding joint labels, the deep network architecture of ResNet-18 is used to extract features and predict joint positions; The constructed dataset is input into the ResNet-18 network to learn the mapping relationship between the voxel projection of point cloud data and the position of human joints; Construct the mean square error (MSE) loss function to measure the error between the joint coordinates predicted by the model and the actual coordinates; in, is the spatial coordinate of the i-th joint predicted by the model, y i is the real coordinate of the i-th joint, and N is the total number of joints.

5. The character modeling method based on non-contact motion capture according to claim 1, wherein: The 3D human body model is trained in the following manner: Use the IK numerical solution algorithm to calculate the root joint and the angle of each joint from the target position. According to the bone structure of the target position, the angle of each joint is calculated by the algorithm. The skeleton structure is used to establish a character model in Blender modeling software through bone connection. The character model is a hierarchical structure consisting of multiple joints and bones. Each joint has an initial position and a rotation angle, and adjacent joints are connected by bones. According to the physical constraints of the human skeleton, the limit range of joint rotation is set, the IK solving algorithm is applied to optimize the Euler angles of the skeleton, the rotation angle of each joint is adjusted to minimize the target error, and the LBFGS optimizer is used to minimize the error between the target position of each pair of joints and the position calculated by the model. By applying L2 regularization to the Euler angles, the overfitting problem of the model is reduced and the posture is kept smooth.

6. A fitness movement teaching method based on character modeling using non-contact motion capture, characterized by: The following steps are involved: S51: Obtaining a standard motion matrix and obtaining a user motion matrix according to the 3D human body model; S52: Calculate the similarity between the standard action matrix and the user action matrix; S53: Determine whether the similarity exceeds a similarity threshold V1 based on the similarity. If it does not exceed the threshold V1, the current action is not performed, and the current action angle matrix is ​​calculated. If it exceeds the threshold V1, determine whether the similarity threshold V2 is reached. If it does not exceed the threshold V2, determine that the action is not standard, and calculate the current action angle matrix. If it exceeds the threshold V2, determine that the action is standard, and proceed to the next step S54. S54: Determine whether there is a next frame action. If yes, import the next frame standard action angle matrix; if not, end the learning; S55: Return to step S51 and continue the cycle.

7. The method for teaching fitness movements based on character modeling using non-contact motion capture according to claim 6, wherein: It also includes fitness action scoring, which is done by following the steps below: S61: extracting the bone rotation angle of the standard motion to form a standard motion data matrix; S62: extracting the skeletal rotation angle of the user's actual movement to form a user action data matrix; S63: Use the DTW algorithm to calculate the similarity between the standard action data matrix and the user action data matrix.

8. The method for teaching fitness movements based on character modeling using non-contact motion capture according to claim 6, wherein: The DTW algorithm is performed as follows: The input data includes user action matrix and standard action matrix; User action matrix: U∈R T×45 ; Standard action matrix: The local distance between the user action matrix and the standard action matrix is ​​calculated according to the following formula: Among them, w k is the joint weight; U i,k,m For the mth Euler angle component of the user's i-th frame and k-th joint, the dynamic programming recursive formula is: Normalized similarity calculation: The user's action scoring rules can be obtained based on the similarity calculation of each frame.

9. The method for teaching fitness movements based on character modeling using non-contact motion capture according to claim 8, wherein: The action scoring rule adopts single-frame scoring or overall scoring; The single-frame scoring formula is: The overall score is the weighted average of all frame scores, with the key frame weight doubled.

10. A fitness movement teaching system for character modeling based on non-contact motion capture, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 6 to 9 is implemented.

Citation Information

Cited By

  • Human body posture reconstruction method based on fusion of MIMO millimeter wave radar and infrared camera

    CN121095381A

  • Human posture reconstruction method based on fusion of MIMO millimeter wave radar and infrared camera

    CN121095381B

  • Intelligent fitness following guidance system and method based on terminal state judgment

    CN122157964A