Three-dimensional human body generation model training method based on single-view-angle shadowboxing action images

Abstract

The invention relates to a training method of a three-dimensional human body generation model based on a single-view-angle shadowboxing action image, and through a single-view-angle-based shadowboxing action analysis method, leading-edge mathematical models and algorithms are introduced, and a highly-vivid three-dimensional digital human model is synchronously generated. According to the method, significant breakthroughs are made in the aspects of the accuracy of motion analysis and the sense of reality of digital human generation, the three-dimensional human body model can be generated without a multi-view camera, the requirements for equipment and sites are reduced, and the accuracy of Tai Chi motion analysis is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a training method, system, computer device and computer-readable storage medium for generating a three-dimensional human body model based on single-view Tai Chi action images. Background Art

[0002] Tai Chi, as a martial arts sport with rich connotations, requires extremely high skills and coordination in every movement. Due to the significant individual differences in body shape, habits and abilities of practitioners, traditional, unchanging standard movements are not always applicable. Therefore, it is particularly urgent to develop a method that can analyze and guide movements based on individual characteristics.

[0003] Traditional two-dimensional image analysis methods have obvious deficiencies in depth and three-dimensional spatial display, and multi-perspective three-dimensional spatial display requires too much hardware equipment, so a portable method that can display multi-dimensional data based on a single perspective is very important. Summary of the invention

[0004] The embodiments of the present application provide a training method, system, computer device and computer-readable storage medium for generating a three-dimensional human body model based on single-view Tai Chi action images, so as to at least solve the problem of poor accuracy in depth and three-dimensional space display of Tai Chi action analysis methods based on two-dimensional images in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a training method for generating a three-dimensional human body model based on a single-view Tai Chi action image, the method comprising:

[0006] Acquire a training data set of Tai Chi movements, and pre-process the training data set to obtain body shape description information, wherein the training data set includes a template data set of standard athletes and an example data set of target users;

[0007] Based on each two-dimensional image in the training data set and its corresponding body shape description information, training a body shape parameter extraction model, wherein the trained body shape parameter extraction model is used to obtain corresponding two-dimensional body shape parameters according to the two-dimensional image;

[0008] Based on each two-dimensional image in the training data set, a joint point reconstruction model is trained, wherein the trained joint point reconstruction model is used to obtain corresponding three-dimensional posture parameters according to the two-dimensional image;

[0009] The two-dimensional body shape parameters and the three-dimensional posture parameters are input into the SMPL model to generate a three-dimensional human body model that is similar to the body shape of the target user and the posture of the standard athlete.

[0010] In some embodiments, preprocessing the training data set to obtain body shape description information includes:

[0011] By analyzing and processing each two-dimensional image in the training data set, the body shape description information of the standard athlete and the target user is obtained respectively, wherein the body shape description information includes: initial posture parameters and initial mesh vertices,

[0012] The initial posture parameters include variables of n×m dimensions, which are used to describe the movement posture of the human body.

[0013] In some embodiments, based on each two-dimensional image in the training data set and its corresponding body shape description information, training the body shape parameter extraction model includes:

[0014] Constructing a neural network model with a two-dimensional image of a preset viewing angle as input and human body shape parameters as output, wherein the neural network model is connected to the SMPL model;

[0015] Based on the two-dimensional image, the neural network model is iteratively optimized and trained, and when the iterative optimization training reaches a preset threshold, a trained body shape parameter extraction model is obtained;

[0016] The neural network model uses the initial mesh vertices corresponding to the two-dimensional image as labels, and determines the loss function based on the difference between the initial mesh vertices and the mesh vertices in the three-dimensional human body model output by the SMPL model.

[0017] In some embodiments, based on each two-dimensional image in the training data set, training the joint point reconstruction model includes:

[0018] Constructing a deep learning model whose input is the two-dimensional image and whose output is the three-dimensional posture parameter;

[0019] The deep learning model is trained by predicting the coordinates of each joint point by regression based on the plurality of the two-dimensional images as a learning task, and when the number of iterations reaches a preset threshold, the trained joint point reconstruction model is obtained;

[0020] The joint reconstruction model extracts visual features from the two-dimensional image through convolutional neural networks and enhances the visual features through a feature pyramid network.

[0021] The enhanced visual features are mapped into three-dimensional joint point coordinates through a posture regression network, and the posture regression network includes multiple fully connected layers, which are used to process the correlation characteristics between the joint points through the interaction of the multiple fully connected layers.

[0022] In some embodiments, the two-dimensional body shape parameters and the three-dimensional posture parameters are input into the SMPL model to generate a three-dimensional human body model similar to the body shape of the target user and similar to the posture of the standard athlete, including:

[0023] Based on the body shape description information of the standard athlete and the target user, respectively, the distances between the joints of the target user and the standard athlete are obtained, and based on the distances between the joints, the scaling ratios between the joints of the standard athlete and the target athlete are determined;

[0024] Scaling the three-dimensional posture parameters of the standard athlete according to each of the scaling ratios to obtain optimized three-dimensional posture parameters;

[0025] A three-dimensional human body model of the target user is generated through the SMPL model according to the optimized three-dimensional posture parameters and the two-dimensional body parameters of the target user.

[0026] In a second aspect, the embodiment of the present application provides a Tai Chi action evaluation method based on a single perspective, the method comprising:

[0027] Based on the Tai Chi action images of the target user, construct a data set to be evaluated, wherein the data set to be evaluated includes a plurality of two-dimensional Tai Chi images and their corresponding action type labels;

[0028] Train the RNN model according to the data set to be evaluated to obtain a Tai Chi action classification model;

[0029] Based on the dataset to be evaluated and the template dataset of a standard athlete, model training is performed by the model training method of claim 1 to obtain a three-dimensional human body model;

[0030] Obtaining an image classification result corresponding to the two-dimensional image based on the data set to be evaluated by using the Tai Chi action classification model;

[0031] Based on the two-dimensional image and its image classification result, as well as the three-dimensional human body model, the Tai Chi movement evaluation result of the user to be evaluated is obtained.

[0032] In some embodiments, based on the two-dimensional image and its image classification result, and the three-dimensional human body model, obtaining the Tai Chi action evaluation result of the user to be evaluated includes:

[0033] Based on the image classification result, obtaining the three-dimensional posture parameters of the user to be evaluated within a target time period corresponding to any Tai Chi movement classification;

[0034] Acquiring reference posture data within the target time period according to the three-dimensional human body model of the user to be evaluated;

[0035] Optimizing the three-dimensional posture parameters of the user to be evaluated through frame sampling or frame interpolation;

[0036] The joint point coordinates of each image frame in the three-dimensional posture parameters of the user to be evaluated after the optimization are compared with the joint point coordinates of each image frame in the reference posture data to obtain the Tai Chi movement evaluation result of the user to be evaluated.

[0037] In a third aspect, an embodiment of the present application provides a training system for generating a three-dimensional human body model based on a single-view Tai Chi action image, the system comprising: an acquisition module, a parameter extraction module, and a human body model generation module;

[0038] The acquisition module is used to acquire a training data set of Tai Chi movements, pre-process the training data set, and obtain body shape description information, wherein the training data set includes a template data set of a standard athlete and an example data set of a target user;

[0039] The parameter extraction module is used to train a body shape parameter extraction model based on each two-dimensional image in the training data set and its corresponding body shape description information, wherein the trained body shape parameter extraction model is used to obtain corresponding two-dimensional body shape parameters according to the two-dimensional image,

[0040] And, based on each two-dimensional image in the training data set, training a joint reconstruction model, wherein the trained joint reconstruction model is used to obtain corresponding three-dimensional posture parameters according to the two-dimensional image;

[0041] The human body three-dimensional model generation module is used to input the two-dimensional body shape parameters and the three-dimensional posture parameters into the SMPL model to generate a human body three-dimensional model that is similar to the body shape of the target user and the posture of the standard athlete.

[0042] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0044] Compared with the related art, the training method for generating a 3D human model based on a single-view Tai Chi action image provided in the embodiment of the present application is based on a single-view Tai Chi action analysis method, which introduces cutting-edge mathematical models and algorithms and simultaneously generates a highly realistic 3D digital human model. This method has made significant breakthroughs in the accuracy of action analysis and the realism of digital human generation. It does not require cameras with multiple perspectives to generate a 3D human model, which reduces the requirements for equipment and venues and improves the accuracy of Tai Chi action analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 It is a flow chart of a training method for generating a three-dimensional human body model based on a single-view Tai Chi action image according to an embodiment of the present application;

[0047] Figure 2 is a flow chart of a Tai Chi action evaluation method based on a single perspective according to an embodiment of the present application;

[0048] Figure 3 It is a structural block diagram of a traffic abnormal behavior adaptive evolution warning method system according to an embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0051] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0052] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] In the analysis of Tai Chi movements, the diversity of human body morphology is a factor that cannot be ignored. For individuals with special body shapes, simply applying standard movement templates (such as the demonstration movements of Tai Chi world champions) often leads to distorted analysis results. In order to solve this problem, the present application constructs a three-dimensional human body generation model that can convert the human body shape of standard athletes into the body shape of users, so as to analyze the movements of Tai Chi under the same body shape.

[0055] In a first aspect, the present application provides a method for training a three-dimensional human body model based on a single-view Tai Chi action image. Figure 1 is a flow chart of a training method for generating a 3D human body model based on a single-view Tai Chi action image according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0056] S101, obtaining a training data set of Tai Chi movements, preprocessing the training data set to obtain body shape description information, wherein the training data set includes a template data set of standard athletes and an example data set of target users;

[0057] Among them, the template data set of standard athletes should contain standardized Tai Chi movements, and the key point data of each movement should be as accurate as possible; and the example data set of target users can be achieved by any Tai Chi enthusiast demonstrating or simulating Tai Chi movements and recording them.

[0058] In addition, data preprocessing is the process of extracting fixed information from raw data by scanning, so as to facilitate subsequent analysis and modeling. In this embodiment, the preprocessing steps specifically include:

[0059] S1, data cleaning

[0060] Missing data processing: Check whether there are missing values ​​in the data set. If there are missing values, measures such as completion or deletion are needed.

[0061] Noise processing: Remove noise from the data, such as erroneous keypoint coordinates or irrelevant action fragments.

[0062] S2, data alignment

[0063] Time alignment: The actions of different users may not be synchronized in time, so time alignment is required. Dynamic Time Warping (DTW) algorithm can be used to align the time series data of actions.

[0064] Spatial alignment: The movements of the standard athlete and the target user may not be consistent in spatial position, so normalization can be performed to make the movement data comparable at a uniform scale. For example, the joint coordinate data can be centered (relative to the center point of the body) or standardized.

[0065] S3, analyze and obtain posture and shape information

[0066] By analyzing each two-dimensional image in the training data set, body shape description information is obtained, wherein the body shape description information includes: initial posture parameters and initial mesh vertices;

[0067] Specifically, a posture estimation algorithm (such as OpenPose, HRNet, etc.) can be used to extract the two-dimensional coordinates of the key points of the human body from the video frame, and output the three-dimensional joint point coordinates of the human body in each frame, that is, the posture parameter θ;

[0068] Among them, the initial posture parameter θ has 24×3 dimensions and is used to describe the movement posture of the human body; among them, the number of joint points is 24, including 23 joint points and 1 root node: each joint point is described by a 3-dimensional variable, and these 3 variables are usually rotation angles (such as Euler angles, quaternions, etc.), which are used to describe the rotation posture of the joint point relative to the root node.

[0069] Furthermore, the root node is the base point of the human body model, and its coordinates are relative to the real position of the entire space. The root node is usually located at the waist or pelvis of the human body, and it is the starting point of the entire skeleton system; other joints except the root node are described by the coordinate system relative to their parent node. The posture (rotation) parameters of each joint point determine the position of the joint point in three-dimensional space.

[0070] Parent-child node relationship: Joints are connected through a parent-child relationship, and each joint inherits the coordinate system of its parent node. For example, if you have a posture parameter for the shoulder joint, it is relative to the root node (usually the pelvis). The position of the forearm joint is relative to the position of the upper arm joint, and so on.

[0071] In addition, if there is 3D scanning data, the vertex coordinates of the mesh can be directly extracted through the 3D mesh processing method to describe the body shape. The initial mesh vertex coordinates obtained by scanning can describe the geometric shape of the human body. These vertices are 3D coordinates and usually form the outer surface of the human body; further, the human body surface is connected by these vertices through polygons (usually triangles or quadrilaterals) to form a 3D mesh model.

[0072] It can be understood that the surface shape and size of the human body can be accurately described through vertex coordinates and meshes. Furthermore, combined with the posture parameter θ, dynamic body shape description and motion simulation can be achieved.

[0073] S102, training a body shape parameter extraction model based on each two-dimensional image in the training data set and its corresponding body shape description information, wherein the trained body shape parameter extraction model is used to obtain corresponding two-dimensional body shape parameters according to the two-dimensional image;

[0074] In this embodiment, the function of this step is to train a model for extracting body shape parameters. Specifically, this step includes the following subdivision steps:

[0075] S1, constructing a neural network model with a two-dimensional image of a preset perspective as input and a human body shape parameter corresponding thereto as output, wherein the neural network model configuration is connected with the SMPL model;

[0076] Wherein, the function of step S1 is to design a neural network model for mapping a two-dimensional image of a human body to three-dimensional body shape parameters;

[0077] In an exemplary embodiment, the neural network model includes the following structure:

[0078] a. Convolutional Neural Network (CNN): CNN is usually used as the backbone network to extract features from two-dimensional images. These features may include contours, joints, muscle distribution, etc.

[0079] b, Fully connected layer: After the convolutional layer, a fully connected layer is used to map the extracted features to the predicted values ​​of the 3D body shape parameters;

[0080] c, regression layer: used to finally output body shape parameters, usually predicting continuous body shape variables through regression.

[0081] S2, based on the two-dimensional image, iteratively optimizes and trains the neural network model, and obtains a trained body shape parameter extraction model when the iterative optimization training reaches a preset threshold;

[0082] Among them, the neural network model uses the initial mesh vertices corresponding to the two-dimensional image as labels, and the difference between the initial mesh vertices and the mesh vertices in the three-dimensional human body model output by the SMPL model as the loss function.

[0083] It can be understood that the loss function is designed as the difference between the initial mesh vertices (estimated based on the 2D image) and the mesh vertices of the 3D model generated by SMPL, which means that the training process aims to minimize the error between the human body shape estimated from the 2D image and the 3D human body shape generated by the SMPL model;

[0084] Optionally, a suitable optimization algorithm, such as Adam or SGD, can be used to minimize the loss function and update the model parameters; by repeatedly iteratively adjusting the weights of the neural network model, the output body shape parameters can generate a three-dimensional model that is closer to the actual shape of the two-dimensional image. When the iterative optimization training reaches a preset threshold (for example, the loss function converges to a certain smaller value), a trained body shape parameter extraction model is obtained.

[0085] In an exemplary embodiment, the training process can divide the data set into a training set, a validation set, and a test set, usually 80% for training, 10% for validation, and 10% for testing. In addition, the model is trained in multiple epochs, and the model weights are updated by the optimizer in each iteration; the validation set is used to evaluate the model performance after each epoch, and the model hyperparameters (such as learning rate, network structure, etc.) are adjusted according to the validation results.

[0086] Through the above step S102, a body parameter extraction model can be trained based on the preprocessed training data set. The model can obtain the body parameters (β) of any standard athlete and target experience user. The body parameters describe individual differences in the human body, such as body shape, height, fatness, etc., which are usually represented by a 10-dimensional vector. The parameters control the basic shape of the human body mesh.

[0087] S103, training a joint reconstruction model based on each two-dimensional image in the training data set, wherein the trained joint reconstruction model is used to obtain corresponding three-dimensional posture parameters according to the two-dimensional image;

[0088] It can be understood that the human body is a rigid object with fixed distances between joints, so three-dimensional data can be fitted based on a two-dimensional image of the human body. In this embodiment, a two-dimensional image of any viewpoint is used as input and a three-dimensional posture parameter is used as output to train the model, so that the model fits the three-dimensional posture parameter based on the two-dimensional image;

[0089] Specifically, this step includes the following subdivision steps:

[0090] S1, build a deep learning model with 2D images as input and 3D posture parameters as output;

[0091] The input data comes from the two-dimensional images collected in step S101, which are usually human images at a certain viewing angle, and contain different postures;

[0092] Furthermore, the label data are the three-dimensional joint point coordinates corresponding to the input image, and these coordinates are also obtained through the preprocessing step in step S101.

[0093] S2, using multiple two-dimensional images to predict the coordinates of each joint point by regression, training the joint point reconstruction model, and obtaining the trained joint point reconstruction model when the number of iterations reaches a preset threshold;

[0094] First, the joint reconstruction model extracts visual features from the two-dimensional image through convolutional neural networks. The convolutional layer can identify edges, contours, and complex shape features in the image.

[0095] Secondly, through the feature pyramid network (FPN) or residual network (ResNet): further extract multi-scale features and enhance the feature expression ability of the model;

[0096] Furthermore, the enhanced visual features are mapped to three-dimensional joint point coordinates through a posture regression network. It should be noted that the posture regression network maps the extracted features to three-dimensional joint point coordinates through a series of fully connected layers or RNNs (such as LSTM); this part of the network needs to process the correlation between joint points and ensure that the output coordinates are reasonable.

[0097] Finally, the three-dimensional joint point coordinates are output through the output layer, and regression is usually used to predict the x, y, and z coordinates of each joint point.

[0098] In addition, a suitable optimization algorithm, such as Adam or RMSprop, can be used to minimize the loss function and optimize the model parameters, wherein the loss function during the model training process can be determined by measuring the difference between the predicted joint point coordinates and the true coordinates.

[0099] Through the above step S103, the joint reconstruction model can be trained based on the preprocessed training data set. The model can obtain the posture parameters (θ) of any standard athlete and target experience user in the entire Tai Chi process. The posture parameters describe the coordinates of each joint point of the human body during the movement and are used to represent the overall posture.

[0100] S104, inputting the two-dimensional body shape parameters and the three-dimensional posture parameters into the SMPL model to generate a three-dimensional human body model that is similar to the body shape of the target user and similar to the posture of a standard athlete.

[0101] Among them, the SMPL (Skinned Multi-Person Linear) model is a 3D human model widely used in computer graphics and computer vision. It can represent various human postures and shapes through a parameterized model, thereby generating 3D human meshes of different postures and body shapes.

[0102] The SMPL model is based on the human skeleton and consists of 24 joint points (including the root node and 23 other joint points), corresponding to the posture parameters mentioned in step S201; secondly, the model uses a three-dimensional mesh with a fixed topology structure to represent the human body, usually consisting of 6890 vertices.

[0103] Furthermore, the SMP model controls the rotation of the human skeleton through the posture parameters (θ), thereby affecting the relative positions of each joint point, and controls the basic shape of the human mesh through the shape parameters (β).

[0104] It should be noted that due to the different body shapes, the body shape parameters and posture parameters obtained under the same action are bound to be different. Therefore, it is necessary to fine-tune the data of standard athletes so that the human body shape obtained through the SMPL model is suitable for the user, so as to provide better and more professional action analysis;

[0105] Based on the above viewpoints, in the present application, the three-dimensional posture information of standard athletes in the Tai Chi process is used and combined with the body shape information of the target user to create a three-dimensional model, thereby obtaining a three-dimensional human body model that is similar to the body shape of the target user and similar to the posture of the standard athlete.

[0106] It should be noted that different posture parameters are obtained in consideration of different body shapes, and the position of joints is relative, that is, the coordinates of each joint are defined relative to its parent joint. This means that if the user and the athlete have different body shapes (for example, different leg lengths), then using the athlete's posture parameters may cause the user's movements to look unnatural or distorted. Therefore, directly using the athlete's posture parameters in combination with the target user's body shape parameters will cause the movements to be deformed.

[0107] In this embodiment, the relative relationship between the joints of the athlete is replaced by the relative relationship between the joints of the user, and each joint is scaled, and the scaling ratio is the ratio of the lengths of the two joints.

[0108] Specifically, based on the body description information of the standard athlete and the target user, the distances between the joints of the target user and the standard athlete are obtained, and multiple joint scaling ratios are determined based on the distances between the joints.

[0109] Furthermore, according to the multiple joint scaling ratios, the three-dimensional posture parameters of the standard athlete are scaled to obtain optimized three-dimensional posture parameters, and the three-dimensional human body model of the target user is obtained using the optimized three-dimensional posture parameters and the two-dimensional body parameters of the target user.

[0110] It can be understood that through the above steps, by utilizing this scaling adjustment mechanism, the target user's movements can be more natural and consistent with the actual body shape, without causing uncoordinated movement performance due to differences in body shape.

[0111] Since the human body 3D model is similar to the specific user in body shape, but the various postures in the Tai Chi action process are collected from standard templates, the human body 3D model can effectively make up for the differences between individuals and create 3D human body models suitable for different body shapes based on images from a single perspective, thus realizing the creation of human body 3D models without the need to use multi-perspective hardware equipment.

[0112] In an exemplary embodiment, a specific implementation method of constructing a three-dimensional human body model through an SMPL model includes the following steps:

[0113] Each body shape constructed by the SMPL model is based on a linear set of base templates. The coordinates of each grid point under different body shape parameters β are set as S n , its dimension is (6890, 3, 10); by linearly superimposing the standard base template, we can get the posture of a human body under the current body shape. The mathematical formula is as follows:

[0114]

[0115] Among them, B S (β) is the obtained linear superposition bias, which represents the influence of each body shape parameter β on the human body grid fixed point; each S n It represents the base mesh deformation under body shape parameters. By superimposing these deformations, the mesh vertices under specific body shape parameters can be obtained to obtain a new human body shape.

[0116] It can be understood that the base template refers to a standard human body mesh shape, and the linear set represents the generation of different body shapes through the linear combination of several predefined deformations (or called "shape vectors").

[0117] Furthermore, the posture parameter θ describes the posture of different joints through the rotation matrix R(θ). The posture adjustment is achieved through the linear evaluation function:

[0118]

[0119] Among them, R n (θ) is the rotation matrix generated by the pose parameters, P n is a weight matrix that defines the effect of posture changes on mesh vertices. Through this process, the mesh deformation based on the user's posture parameters can be obtained;

[0120] Each human body posture parameter is represented by a rotation matrix R, k=23 is the number of joints, the axis angle is 3, the rotation dimension of all axis angles is 3K, and the dimension after conversion to the rotation matrix is ​​9K. p is the weight matrix with a dimension of (6890, 3, 207). Like the human body posture, the linear superposition deviation in a base state is obtained. The actual posture is also obtained by superimposing the base posture.

[0121] Specifically, the axis-angle rotation formula is:

[0122]

[0123] Among them, w j is the axis angle.

[0124] In addition, when the human body moves, the network vertices will move with the movement of the joints, and the linear skinning algorithm is needed to perform linear weighted combination of the mesh vertices.

[0125] T=T0+B s (β)+B p (θ)

[0126] Among them, T| is the average mesh vertex of the human body, and T0| is the human body mesh vertex of the base template.

[0127] Through the linear skinning algorithm, the mesh vertices can be further adjusted to adapt to the movement of the joints, and the regression matrix F between the network vertices and the 24 joint points and the hybrid weight matrix W of the linear skinning can be obtained, so that the final relationship can be obtained:

[0128] M(β, θ)=W(T(β, θ), F(T+B s (β), θ, W0)

[0129] This relationship combines the regression matrix F of mesh vertices and joint points and the mixing weight matrix W to generate the final 3D human body model.

[0130] Through the above step S104, a three-dimensional human body model that is consistent with the user's body shape and the standard athlete's movement posture can be constructed to ensure that the athlete's movements are performed more naturally and accurately on the user.

[0131] Through the above steps S101 to S104, compared with the methods in traditional technologies, the problem of insufficient depth sense exists in analyzing Tai Chi movements based on two-dimensional images, and the three-dimensional motion capture system often requires complex hardware support. The present application uses a single-view Tai Chi movement image as input, and uses an innovative algorithm to generate a three-dimensional digital human in a three-dimensional perspective while maintaining the low cost and portability of the device, providing accurate Tai Chi movement analysis; the method extracts the joint position information in the two-dimensional image, and matches the standard movements of the Tai Chi world champion with the user's body shape based on the SMPL (Skinned Multi-Person Linear) model, so that the movement analysis is performed in the same body shape, thereby avoiding movement errors caused by body shape differences; the analysis algorithm can be automatically adjusted according to the user's body shape, habits and abilities to provide more personalized guidance.

[0132] In a second aspect, the present application also provides a Tai Chi action evaluation method based on a single perspective. Figure 2 is a flow chart of a Tai Chi action evaluation method based on a single perspective according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0133] S201, constructing a dataset to be evaluated based on the Tai Chi action images of the target user, wherein the dataset to be evaluated includes a plurality of two-dimensional Tai Chi images and their corresponding action type labels;

[0134] The target user is the target user in the above steps S101 to S103. The target user's body shape, gender, and familiarity with Tai Chi movements have no effect on the present application solution and are not specifically limited in this embodiment.

[0135] S202, training the RNN model according to the data set to be evaluated to obtain a Tai Chi action classification model; performing model training based on the data set to be evaluated and a template data set of a standard athlete by the model training method provided in the above embodiment to obtain a three-dimensional human body model;

[0136] The input of the model is two-dimensional image data of continuous frames, and the output is the confidence of 25 action categories corresponding to each frame. Specifically, 24 are standard Tai Chi movements and 1 is a non-Tai Chi movement.

[0137] In this embodiment, an RNN model is used to process time series data. Since image data usually requires high-dimensional feature extraction, a CNN (convolutional neural network) is used to first extract the features of each frame, and then the feature sequence is input into the RNN (such as LSTM or GRU).

[0138] The network structure of the convolutional neural network may include: a CNN layer for extracting features of each frame of the image; an RNN layer for processing the extracted feature sequence; and a fully connected layer for outputting the classification confidence of the action.

[0139] When the model recognizes an action, the timestamp of the current frame is recorded as the end time of the action, and the start time of the action can be set to the timestamp of the first frame stored in X. In addition, if a more accurate time point is required, consider adjusting the start time through a sliding window technique or a confidence-based threshold.

[0140] Finally, the specific process of generating the 3D human body model of the target user is the same as the above steps S101 to S103, which will not be repeated here.

[0141] S203, obtaining an image classification result corresponding to the two-dimensional image based on the data set to be evaluated by using the Tai Chi movement classification model;

[0142] This step is related to the model building in step S202. It can be understood that the primary task of the action classification model is to identify the specific action that the user is performing. Since Tai Chi contains multiple standard actions, different actions have different postures and action trajectories. By classifying the actions, the system can determine which action the user is currently performing.

[0143] Only by accurately identifying the type of action performed by the user can the system conduct subsequent action evaluation and feedback. This is also an important prerequisite for evaluating the standardization of user actions, dividing the time period corresponding to each action, and providing data support for selecting image frames for comparison in this time period;

[0144] That is, the classification model can help the system understand the specific type of user actions, and based on the classification results, extract the corresponding standard action data for comparison. In action evaluation, the system needs to compare a certain category of user actions with standard actions. If the classification is inaccurate, the user's action may be compared with the wrong standard action, resulting in inaccurate evaluation results. Therefore, the accuracy of action classification is directly related to the accuracy of action evaluation.

[0145] In addition, the action classification model can also distinguish between Tai Chi and non-Tai Chi actions. For example, when the user performs other irrelevant exercises, the system can identify and ignore these non-Tai Chi actions through the classification model. This helps filter out irrelevant action data, ensuring that the system only evaluates Tai Chi-related actions, improving the efficiency and accuracy of the system.

[0146] S204, obtaining Tai Chi movement evaluation results of the user to be evaluated based on the Tai Chi movement image set and its corresponding image classification results, as well as the three-dimensional human body model.

[0147] Specifically, step S205 includes the following subdivision steps:

[0148] Step 1, based on the image classification results, obtain the three-dimensional posture parameters of the user to be evaluated within the target time period corresponding to any Tai Chi action classification;

[0149] Step 2, obtaining reference posture data within the target time period based on the 3D human body model of the user to be evaluated;

[0150] Step 3, optimize the 3D posture parameters of the user to be evaluated through frame sampling or frame interpolation;

[0151] Step 4, by comparing the joint point coordinates of each image frame in the three-dimensional posture parameters of the user to be evaluated after optimization with the joint point coordinates of each image frame in the reference three-dimensional posture parameters, the Tai Chi movement evaluation result of the user to be evaluated is obtained.

[0152] Through the above steps Step 1-Step 3, based on the start time and end time recorded for each action category, the data of the corresponding time period is extracted from the joint point data of the target user. However, since the start time of the action may not be accurate, the start time can be corrected by moving the previous and next frames.

[0153] In addition, the joint point data within the time period is sampled or interpolated to make its length consistent with the standard action duration; assuming that the athlete's standard action within the time period consists of 50 frames of joint point data, but the user's actual action consists of 40 frames of joint point data. To compare the standardization of the two actions, the user's 40 frames of data need to be adjusted to 50 frames, so that the user and athlete's actions can be compared frame by frame.

[0154] If the user's frame number is more than the frame number of the standard action, it is necessary to evenly select some frames from the user's frame data to reduce the data length; if the user's frame number is less than the frame number of the standard action, it may be necessary to delete some frames from the standard action, or interpolate and expand the user's data to make its frame number consistent with the standard action.

[0155] If the user's action has fewer frames than the standard action, an interpolation method can be used to generate additional frames so that the user's action sequence length is consistent with the standard action. Interpolation can be achieved through linear interpolation or more complex temporal interpolation methods.

[0156] Through this adjustment and optimization, the user's joint point data and the athlete's standard movement data will have the same time length, which is convenient for frame-by-frame comparison and difference analysis, and then judgement of the user's movement standardization.

[0157] Through the above steps S201 to S203, the user's Tai Chi movements are evaluated to see whether they meet the standards through precise three-dimensional posture comparison. This evaluation is based on the dual dimensions of time and space. By comparing the coordinates of the joint points frame by frame, it can accurately capture the subtle deviations in the user's movements and help the user continuously optimize the standardization of the movements during practice. Ultimately, the system provides users with a data-driven, precise movement evaluation and improvement tool that helps improve the standardization and effectiveness of Tai Chi movements.

[0158] In a second aspect, this embodiment also provides a training system for generating a 3D human body model based on single-view Tai Chi action images. Figure 3 is a structural block diagram of a traffic abnormal behavior adaptive evolution warning method system according to an embodiment of the present application, such as Figure 3 As shown, the system includes: an acquisition module 30, a parameter extraction module 31 and a human body model generation module 32;

[0159] The acquisition module 30 is used to acquire a training data set of Tai Chi movements, pre-process the training data set, and obtain body shape description information, wherein the training data set includes a template data set of a standard athlete and an example data set of a target user;

[0160] The parameter extraction module 31 is used to train a body shape parameter extraction model based on each two-dimensional image in the training data set and its corresponding body shape description information, wherein the trained body shape parameter extraction model is used to obtain corresponding two-dimensional body shape parameters according to the two-dimensional image, and to train a joint point reconstruction model based on each two-dimensional image in the training data set, wherein the trained joint point reconstruction model is used to obtain corresponding three-dimensional posture parameters according to the two-dimensional image;

[0161] The human body three-dimensional model generation module 21 is used to input two-dimensional body shape parameters and three-dimensional posture parameters into the SMPL model to generate a human body three-dimensional model that is similar to the body shape of the target user and similar to the posture of a standard athlete.

[0162] Through the above system, compared with the traditional technology, the analysis of Tai Chi movements based on two-dimensional images has the problem of insufficient depth, and the three-dimensional motion capture system often requires complex hardware support. This system uses Tai Chi movement images from a single perspective as input, and uses innovative algorithms to generate a three-dimensional digital human under a three-dimensional perspective while maintaining the low cost and portability of the equipment, providing accurate Tai Chi movement analysis; this method extracts joint position information from two-dimensional images, based on the SMPL (Skinned Multi-Person Linear) model, through computer graphics, deep neural networks, and motion analysis technology, to achieve accurate analysis and digital reproduction of Tai Chi movements; the generated three-dimensional digital human model has a high degree of authenticity and fidelity, and can be applied to the research, teaching and display of Tai Chi movements. .

[0163] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0164] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0165] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0166] S1, obtaining a training data set of Tai Chi movements, preprocessing the training data set, and obtaining body shape description information;

[0167] S2, training a body shape parameter extraction model based on each two-dimensional image in the training data set and its corresponding body shape description information;

[0168] S3, training the joint reconstruction model based on each two-dimensional image in the training data set;

[0169] S4, inputting the two-dimensional body shape parameters and the three-dimensional posture parameters into the SMPL model to generate a three-dimensional human body model that is similar to the body shape of the target user and the posture of a standard athlete.

[0170] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0171] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 4 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 4As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system, the computer program is executed by the processor to implement a training method for generating a three-dimensional human body model based on a single-view Tai Chi action image, and the database is used to store data.

[0172] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. Specifically, the electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0173] In addition, in combination with the adaptive evolution warning method for abnormal traffic behavior in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the training methods for generating a three-dimensional human body model based on a single-view Tai Chi action image in the above embodiment is implemented.

[0174] In one embodiment, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, any one of the adaptive evolution warning methods for abnormal traffic behavior in the above embodiments is implemented.

[0175] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0176] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A training method for generating a three-dimensional human body model based on single-view Tai Chi action images, characterized in that: The method comprises: Acquire a training data set of Tai Chi movements, and preprocess the training data set to obtain body shape description information, wherein the training data set includes a template data set of standard athletes and an example data set of target users; Based on each two-dimensional image in the training data set and its corresponding body shape description information, training a body shape parameter extraction model, wherein the trained body shape parameter extraction model is used to obtain corresponding two-dimensional body shape parameters according to the two-dimensional image; Based on each two-dimensional image in the training data set, training a joint point reconstruction model, wherein the trained joint point reconstruction model is used to obtain corresponding three-dimensional posture parameters according to the two-dimensional image; The two-dimensional body shape parameters and the three-dimensional posture parameters are input into the SMPL model to generate a three-dimensional human body model that is similar to the body shape of the target user and the posture of the standard athlete.

2. The method according to claim 1, characterized in that Preprocessing the training data set to obtain body shape description information includes: By analyzing and processing each two-dimensional image in the training data set, the body shape description information of the standard athlete and the target user is obtained respectively, wherein the body shape description information includes: initial posture parameters and initial mesh vertices, The initial posture parameters include variables of n×m dimensions, which are used to describe the movement posture of the human body.

3. The method according to claim 2, characterized in that According to each two-dimensional image in the training data set and its corresponding body shape description information, training a body shape parameter extraction model includes: Constructing a neural network model with a two-dimensional image of a preset viewing angle as input and human body shape parameters as output, wherein the neural network model is connected to the SMPL model; Based on the two-dimensional image, the neural network model is iteratively optimized and trained, and when the iterative optimization training reaches a preset threshold, a trained body shape parameter extraction model is obtained; The neural network model uses the initial mesh vertices corresponding to the two-dimensional image as labels, and determines the loss function based on the difference between the initial mesh vertices and the mesh vertices in the three-dimensional human body model output by the SMPL model.

4. The method according to claim 1, characterized in that: Based on each two-dimensional image in the training data set, training the joint point reconstruction model includes: Constructing a deep learning model whose input is the two-dimensional image and whose output is the three-dimensional posture parameter; The deep learning model is trained by predicting the coordinates of each joint point by regression based on the plurality of the two-dimensional images as a learning task, and when the number of iterations reaches a preset threshold, the trained joint point reconstruction model is obtained; The joint reconstruction model extracts visual features from the two-dimensional image through convolutional neural networks and enhances the visual features through a feature pyramid network. The enhanced visual features are mapped into three-dimensional joint point coordinates through a posture regression network, and the posture regression network includes multiple fully connected layers, which are used to process the correlation characteristics between the joint points through the interaction of the multiple fully connected layers.

5. The method according to claim 1, characterized in that Inputting the two-dimensional body shape parameters and the three-dimensional posture parameters into the SMPL model to generate a three-dimensional human body model similar to the body shape of the target user and similar to the posture of the standard athlete includes: Based on the body shape description information of the standard athlete and the target user, respectively, the distances between the joints of the target user and the standard athlete are obtained, and based on the distances between the joints, the scaling ratios between the joints of the standard athlete and the target athlete are determined; Scaling the three-dimensional posture parameters of the standard athlete according to each of the scaling ratios to obtain optimized three-dimensional posture parameters; A three-dimensional human body model of the target user is generated through the SMPL model according to the optimized three-dimensional posture parameters and the two-dimensional body parameters of the target user.

6. A Tai Chi action evaluation method based on a single perspective, characterized in that: The method comprises: Based on the Tai Chi action images of the target user, construct a data set to be evaluated, wherein the data set to be evaluated includes a plurality of two-dimensional Tai Chi images and their corresponding action type labels; Train the RNN model according to the data set to be evaluated to obtain a Tai Chi action classification model; Based on the dataset to be evaluated and the template dataset of a standard athlete, model training is performed by the model training method of claim 1 to obtain a three-dimensional human body model; Obtaining an image classification result corresponding to the two-dimensional image based on the data set to be evaluated by using the Tai Chi action classification model; Based on the two-dimensional image and its image classification result, as well as the three-dimensional human body model, the Tai Chi movement evaluation result of the user to be evaluated is obtained.

7. The method according to claim 6, characterized in that According to the two-dimensional image and its image classification result, and the three-dimensional human body model, the Tai Chi action evaluation result of the user to be evaluated is obtained, including: Based on the image classification result, obtaining the three-dimensional posture parameters of the user to be evaluated within a target time period corresponding to any Tai Chi movement classification; Acquiring reference posture data within the target time period according to the three-dimensional human body model of the user to be evaluated; Optimizing the three-dimensional posture parameters of the user to be evaluated through frame sampling or frame interpolation; The joint point coordinates of each image frame in the three-dimensional posture parameters of the user to be evaluated after the optimization are compared with the joint point coordinates of each image frame in the reference posture data to obtain the Tai Chi movement evaluation result of the user to be evaluated.

8. A training system for generating a three-dimensional human body model based on single-view Tai Chi action images, characterized in that: The system comprises: an acquisition module, a parameter extraction module and a human body model generation module; The acquisition module is used to acquire a training data set of Tai Chi movements, pre-process the training data set, and obtain body shape description information, wherein the training data set includes a template data set of a standard athlete and an example data set of a target user; The parameter extraction module is used to train a body shape parameter extraction model based on each two-dimensional image in the training data set and its corresponding body shape description information, wherein the trained body shape parameter extraction model is used to obtain corresponding two-dimensional body shape parameters according to the two-dimensional image, And, based on each two-dimensional image in the training data set, training a joint point reconstruction model, wherein the trained joint point reconstruction model is used to obtain corresponding three-dimensional posture parameters according to the two-dimensional image; The human body three-dimensional model generation module is used to input the two-dimensional body shape parameters and the three-dimensional posture parameters into the SMPL model to generate a human body three-dimensional model that is similar to the body shape of the target user and the posture of the standard athlete.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images