A human model neural network training method, device and storage medium

Abstract

The application discloses a human model neural network training method, comprising the following steps: obtaining a two-dimensional image of a target human body; constructing a three-dimensional standard human model; obtaining three-dimensional human body parameters of a target human model through a neural network; obtaining a three-dimensional target human model grid which is the same as the posture and body shape of the target human body; importing the two-dimensional image and the target human model into three-dimensional modeling software; adjusting the two-dimensional image and the target human model for consistency by using the three-dimensional modeling software; and inputting a new target human model as an output result into the neural network to complete the training of the neural network. The three-dimensional human model neural network training method provided by the application can greatly improve the parameter precision of the deep learning neural network and significantly accelerate the convergence speed by aligning and adjusting the details of the generated human model, and can greatly improve the consistency and restoration degree of the three-dimensional human model generation.

Description

Technical Field

[0001] This invention belongs to the field of human body 3D modeling, specifically relating to a training method, device and storage medium for a neural network model used in human body modeling, and in particular a method for training a neural network model by modifying a human body model generated by a neural network model and then using the modified model. Background Technology

[0002] With the development of internet technology, online shopping has become increasingly popular. Compared to shopping in physical stores, online shopping offers advantages such as a wider variety of goods and greater convenience. However, buying goods online also presents some unresolved problems, the most significant being the inability to physically view the items being purchased. This issue is most pronounced for clothing. Unlike physical stores where customers can try on clothes and see how they look, online clothing stores cannot provide personalized images; they only offer pictures of models trying on clothes, and some online stores don't even offer any pictures of models trying on clothes. Consumers cannot directly assess how well the clothing matches their own body type and appearance, leading to a large number of returns and exchanges.

[0003] To address this issue, businesses have attempted to use virtual try-on technology to provide consumers with a simulated fitting experience. Of course, there are other real-world applications where virtual try-on technology can be used, such as in online games. Therefore, this technology has experienced rapid development.

[0004] Virtual try-on refers to a technology that allows users to view the "fitted" effect in real-time on a terminal screen without actually trying on the clothes they want to see. Existing virtual try-on technologies mainly include 2D virtual try-on and 3D virtual try-on. The former basically involves collecting images of the user and the clothing, then cropping and stitching them together to create an image of the "dressed" person. However, this type of image suffers from poor realism due to its simplistic image processing methods, completely disregarding the user's actual body shape and merely applying the clothing to a photo, failing to meet user needs. The latter typically uses 3D acquisition equipment to collect 3D information about the person and combines it with clothing features for synthesis, or it involves manually inputting body data provided by the user and generating a virtual 3D human model according to certain rules, which is then combined with clothing textures. Overall, this type of 3D virtual try-on requires extensive data collection or 3D data calculation, resulting in high hardware costs and making it difficult to promote among ordinary users.

[0005] With the development of cloud computing, artificial intelligence, and smart terminal processing capabilities, two-dimensional virtual fitting technology has emerged. This technology mainly includes three steps: (1) processing the personal body information provided by the user to obtain a target human body model; (2) processing the clothing information to obtain a clothing model; and (3) merging the human body model and the clothing model together to generate a simulation image of a person wearing the clothing.

[0006] However, due to the accumulation of many uncertainties such as process design, model parameter selection, and neural network training methods, the quality of the final generated clothing images is not as good as that of traditional 3D virtual try-on technology. Among these, the establishment of the human body model is the basic step, and the subsequent dressing process must also be based on the human body model generated earlier. Therefore, if the human body model is not generated accurately, it is easy to produce problems such as a large difference between the human body model and the body shape of the try-on person, loss of skin texture, and loss of body parts, which will affect the final effect of the generated clothing images.

[0007] In the general field of computer vision, there are many starting points for human body modeling, typically including three main categories: omnidirectional scanning of a real human body using 3D scanning equipment; 3D reconstruction methods based on multi-view depth photography; and methods combining given images with a human body model to achieve 3D reconstruction. Among these, omnidirectional scanning of a real human body using 3D scanning equipment yields the most information and is the most accurate. However, such equipment is usually expensive and requires a high degree of cooperation from the human model. The entire processing process places very high demands on the processing equipment, so it is generally used in specialized fields. Secondly, multi-view 3D reconstruction methods require providing overlapping images of the reconstructed human body from multiple perspectives and establishing spatial transformation relationships between the images. Using multiple cameras to capture multiple images and then stitching them together to create a 3D model simplifies the operation somewhat, but the computational complexity remains high. Furthermore, in most cases, only people present at the scene can obtain multi-angle images. The model obtained by stitching together textures from multi-angle depth camera photography lacks body-scale data and cannot provide a foundation for 3D perception. Secondly, the method combining a single image with a human body model only requires one image. The intelligent generation method of 3D human body feature curves based on neural networks, through neural network training, obtains weights and thresholds that can describe curves of parts of the human body such as the neck, chest, waist, and hips. Then, based on the dimensional parameters such as the circumference, width, and thickness of the human body cross-section, it can directly generate a 3D human body curve that matches the real human body shape, thus obtaining a predicted human body model. However, this method suffers from limited input information and still requires a significant amount of computation, resulting in unsatisfactory final model performance.

[0008] Given the characteristics of internet technology and the network environment, directly outputting the final human body model from a single image is undoubtedly the preferred method, offering the greatest convenience. Users don't need to be physically present; a single photograph is sufficient to complete the entire clothing-changing process. The ensuing question is: if the resulting photograph can be guaranteed to be roughly equivalent to a realistic 3D simulation of clothing changing, it will become the mainstream approach. Therefore, the crucial question becomes: how to obtain a human body model that most closely resembles the real human form from a single photograph?

[0009] In the existing technology, there are usually several types of methods for constructing human body models: (1) Regression-based methods, which reconstruct the human body model represented by voxels through convolutional neural networks. The algorithm first estimates the position of the main joints of the human body based on the input image, and then estimates the position of the key points. In a given voxel grid of a specified size, the shape of the reconstructed human body is described by the shape of the entire voxel occupied by the voxel. (2) Human body reconstruction based on a single image. This method estimates the three-dimensional shape and posture of the human body at the same time. This method first roughly marks the simple human skeleton key points on the image, and then performs initial matching and fitting of the human body model based on these coarse key points to obtain the approximate shape of the human body. (3) Use 23 bone nodes to represent the human skeleton, and then use the rotation of each bone node to represent the posture of the entire human body. At the same time, use 6890 vertex positions to express the human body shape. In the fitting process, given the bone node positions, the parameters of shape and posture are fitted at the same time to perform three-dimensional human body reconstruction. Alternatively, a CNN model is used to predict the key points on the image, and then an SMPL model is used for fitting to obtain the initial human body model. Next, the fitted shape parameters are used to regress a human joint bounding box, with each joint corresponding to a bounding box, represented by its axis length and radius. Finally, the initial model and the regressed bounding boxes are combined to obtain a 3D human reconstruction. The above method suffers from slow modeling speed, insufficient modeling accuracy, and a strong dependence on the created body and pose database for reconstruction results.

[0010] Existing technology 1 discloses a human body modeling method based on body measurement data, such as Figure 1As shown, the method includes: acquiring body measurement data; performing linear regression on a pre-created human body model using a pre-trained prediction model based on the body measurement data to obtain a predicted human body model, wherein the pre-created human body model includes multiple predefined sets of labeled feature points and corresponding standard shape bases, and the body measurement data includes measurement data corresponding to each set of labeled feature points; and obtaining a target human body model based on the predicted human body model, wherein the target human body model includes measurement data, a target shape base, and a target shape coefficient. However, this method has very high requirements for body measurement data, including body length and circumference data, such as height, arm length, shoulder width, leg length, calf length, thigh length, foot length, head circumference, chest circumference, waist circumference, and thigh circumference, etc., which not only requires measurement but also calculation. While it does save computation, the user experience is very poor, and the program is cumbersome. Furthermore, the training method of the SMPL model is referenced in the training of the human body model.

[0011] The SMPL model is a parametric human body model, a human body modeling method proposed by the Max Planck Institute. This method can perform arbitrary human body modeling and animation-driven operations. The biggest difference between this method and traditional LBS (Location-Based Services) lies in its proposed method of human posture image surface morphology. This method can simulate the bulges and depressions of muscles during limb movement. Therefore, it can avoid surface distortion of the human body during movement and accurately depict the morphology of muscle stretching and contraction movements. In this method, β and θ are the input parameters. β represents 10 parameters related to individual body height, weight, head-to-body ratio, etc., while θ represents 75 parameters representing the overall human body posture and the relative angles of 24 joints. However, the core of this model generation method is the accumulation of a large amount of training data to obtain the relationship between body shape and shape basis. However, due to the strong interrelationships between these relationships, each shape basis cannot be independently controlled, making decoupling operations difficult. For example, there is also a certain correlation between arms and legs; when the arm moves, the leg should theoretically move as well. It is difficult to achieve improvements for different body shapes using the SMPL model.

[0012] Existing technology 2 discloses a method for generating a three-dimensional human body model, including: acquiring a two-dimensional human body image; inputting the two-dimensional human body image into a three-dimensional standard human body model to obtain three-dimensional human body parameters corresponding to the two-dimensional human body image; inputting the training samples into a neural network for training to obtain a three-dimensional standard human body model, including: inputting the standard two-dimensional human body image in the training samples into the neural network to obtain predicted three-dimensional human body parameters corresponding to the standard two-dimensional human body image; adjusting a three-dimensional flexible deformable model according to the predicted three-dimensional human body parameters to obtain a predicted three-dimensional human body model; and obtaining the predicted joint point positions in the standard two-dimensional human body image through back mapping based on the joint point positions in the predicted three-dimensional human body model. This modeling method utilizes model judgment and the parameters output by the neural network, which only relate to node parameters. It then uses the mature body shape of the SMPL model to perform detailed adjustments to match the target human body posture. Although this reduces the computational load, due to the limited input parameters and the fact that adjustments can only be made based on the SMPL prediction model, it is difficult to output a human body model that is highly consistent with the target human body posture.

[0013] The prior art 3 discloses a 3D virtual fitting platform based on machine learning, including: (1) the user inputs height, weight, bust, waist, hip, and skin color, and the mobile phone camera captures the user's photo information to the 3D virtual fitting platform; (2) the 3D virtual fitting platform uses machine learning technology to train based on a large amount of real data and models on the platform, and matches the model closest to the user's body shape according to the user's input height, weight, bust, waist, hip, and skin color information; (3) after the user selects clothes online, clicks to try them on, and jumps to the 3D virtual fitting platform. The platform uses 3ds Max technology to match and model the clothes with the human body model and generate an actual effect model. A convolutional neural network is established using the TensorFlow framework, with the user's input height, weight, bust, waist, hip, and face shape as the x vector, and is trained using existing real sample data; after a large amount of calculation, data including the model's shoulder width, upper and lower arm thickness, upper and lower arm length, upper and lower leg thickness, upper and lower leg length, user-input chest circumference, and user-input hip circumference are obtained, and these data are used as variables in combination with 3ds Max technology for 3D modeling. The biggest drawback of this technology is that it is very difficult to obtain data from real human bodies, requiring actual measurement of every real sample, which is a very uneconomical workload.

[0014] In the current field of human body modeling, the method of generating human body models using neural network models is relatively mainstream due to the availability of massive amounts of training images and the relatively good performance of the output models. Therefore, the quality of the generated model, that is, the degree of similarity to the original image, has become the most critical factor. For deep learning neural networks, how to train them and achieve optimal training results and efficiency has become an urgent problem to be solved. Summary of the Invention

[0015] To align with the development trends of the internet industry, in the niche field of virtual try-on, especially human body modeling, the three fundamental goals that have always been pursued are minimal input information, minimal computational load, and optimal results. We need to find an optimal balance among these three, providing a human body modeling method that allows for simple input, computational loads not exceeding the capacity of terminal devices, and results approaching those of professional equipment. To address these issues, this invention provides a method, device, and storage medium for training a neural network for human body models that overcomes the aforementioned problems.

[0016] This invention provides a method for training a neural network for a human body model, the method comprising:

[0017] 1) Obtain a two-dimensional image of the target human body;

[0018] 2) Obtain the parameters of the target human body model using a neural network, and then obtain a 3D target human body model mesh based on the parameters;

[0019] 3) Import the 2D image and target human body model into the 3D modeling software;

[0020] 4) Use 3D modeling software to make consistency adjustments between the 2D image and the target human body model to obtain a corrected new target human body model;

[0021] 5) Substitute the new target human body model as the output into the neural network for calculation;

[0022] 6) Use the calculation results to adjust the model parameters and complete the training of the neural network.

[0023] The training method further includes: constructing a three-dimensional standard human body model by combining a mathematical model; obtaining three-dimensional human body parameters of the target human body model through a neural network, including three-dimensional human body posture parameters and three-dimensional human body shape parameters; corresponding the obtained three-dimensional human body posture and body shape parameters with several basis and skeletal parameters of the three-dimensional standard human body model; and inputting the obtained several sets of basis and skeletal parameters into the standard three-dimensional standard human body model for fitting.

[0024] Preferably, the step of obtaining the parameters of the target human body model further includes: 1) obtaining a two-dimensional image of the target human body; 2) processing to obtain a two-dimensional human body contour image of the target human body; 3) substituting the two-dimensional human body contour image into a first neural network after deep learning to perform joint point regression; 4) obtaining a joint point map of the target human body; obtaining semantic segmentation maps of various parts of the human body; body key points; body skeletal points; 5) substituting the generated joint point map, semantic segmentation map, body skeletal points and key point information of the target human body into a second neural network after deep learning to perform regression of human posture and body shape parameters; 6) obtaining the output three-dimensional human body parameters, including three-dimensional human body action posture parameters and three-dimensional human body shape parameters.

[0025] Preferably, before inputting the two-dimensional human body image into the first neural network model, the process of training the first neural network is also included. The training samples include standard two-dimensional human body images with original joint point positions marked, wherein the original joint point positions are marked with high accuracy by humans on the two-dimensional human body images.

[0026] Preferably, the three-dimensional standard human body model consists of parameters of several body bases and several bone parameters. These body bases constitute the entire human body model, and each body base is individually controlled by its parameters, without affecting the others. The three-dimensional standard human body model has a mathematical weight relationship between bone points and model meshes, and the determination of bone points can be associated with the determination of the human body model of the target human posture.

[0027] Preferably, the consistency adjustment is performed manually. The consistency of posture between the imported target human body model and the two-dimensional image of the target human body is achieved by adjusting the skeleton of the human body model, and the consistency of body shape is achieved by adjusting the shape base of the human body model. After obtaining the corrected new target human body model, the three views of the human body model are rendered in the software, compared with the photograph, and then further corrected and adjusted.

[0028] A computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the steps described above.

[0029] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements any of the steps described above.

[0030] The beneficial effects of this invention are:

[0031] 1. Excellent Training Results. This is because we train the neural network with the most accurate results. As we all know, there are numerous neural network algorithms and parameters, and loss functions vary depending on project requirements. Only through continuous training and iteration can the network approach the most accurate "answer." In actual testing, we found that for neural networks that directly output 3D human body models from a single image, the accuracy is not very high due to the influence of various parameters and LOS functions, especially for complex human poses. However, current technology lacks effective methods to address these errors. Substituting the output results back into the model results in slow convergence, while directly adjusting parameters is highly unpredictable and has fluctuating convergence speed. Our approach involves importing the target human body model generated by the neural network (which may not accurately reflect the true posture and body shape of the human in the photo) into mainstream 3D modeling software, while simultaneously importing the original photo into the software. We then manipulate the human body model to visually "fit" the photo, resulting in a corrected new target human body model. This new model is much closer to the true human posture and body shape in the photo than the directly output model. Then, this corrected model is substituted into the neural network for training. This is equivalent to training the neural network with the correct answer, which can greatly improve the accuracy of the parameters and the rationality of the loss. The neural network model has a very ideal convergence speed, and after training, a target human body model with a high degree of consistency with the pose and body shape of the human body photo can be obtained.

[0032] 2. The neural network model is set up more scientifically and purposefully. Some existing image processing methods overemphasize simply producing a model directly without spending time refining its details. They rely solely on training massive amounts of image data to map 2D images to 3D body models. While this is highly efficient, the processing flow is too simplistic. Relying entirely on neural network models to generate 3D human models results in unsatisfactory consistency and quality in body proportions and details. Furthermore, it offers no help for subsequent processing and may even become a major obstacle for later stages. In contrast, we can take these less-than-ideal human models, make slight adjustments, and use them as a good starting point to train neural networks. By adjusting parameters, we can gradually approach the ideal output state of the neural network.

[0033] 3. High-frequency use of deep neural networks. This invention fully utilizes the advantages of deep learning networks, enabling high-precision reconstruction of human posture and body shape in various complex scenarios. Different neural networks are used for different purposes, and neural network models with different input conditions and training methods are employed to achieve accurate contour separation, semantic segmentation, and key point and joint point determination of the human body in complex backgrounds. This eliminates the influence of loose clothing and hairstyles, achieving the closest possible approximation to the real human body shape and form. Existing technologies also use neural network models, but due to differences in input conditions, input parameters, and training methods, the functions and roles of the neural network models vary significantly. In our first-stage neural network, human contour, semantic segmentation, key points, and joint points are all taken as input items, allowing for the generation of model parameters from multiple perspectives. Furthermore, the output parameters of the second-stage neural network include both posture and body shape categories, which can control movement and body shape separately. Combined with our reference model, it can accurately replicate the posture and body shape of the human model.

[0034] 4. Precise and Controllable Human Body Model. Currently popular single-image-based human body reconstruction methods mainly fall into two categories: reconstructing parametric human body models. The most commonly used parametric model is the Max Planck Institute's SMPL model, which contains two sets of 72 parameters describing human posture and body shape. For single-image reconstruction, the SMPL parameters are first estimated from the image, and then optimized by minimizing the projection distance between the 3D joints and the 2D planar joints, thus obtaining the human body. However, the SMPL model is mainly trained through a large number of human body model instances using deep learning. The relationship between body shape and shape basis is a holistic one, making decoupling very difficult. It is impossible to control the desired body parts at will, resulting in a model that cannot achieve a high degree of consistency with the real human posture and body shape. Furthermore, if it is further applied to the subsequent dressing process, it will also have limited ability to represent the geometric details of the human body surface, and cannot reconstruct the detailed texture of clothing on the human body surface well. However, our human body model is not obtained through training. The parameters have a mathematically based correspondence, meaning that our different sets of parameters are independent and not interconnected. Therefore, our model is more interpretable during transformations and can better represent shape changes in a specific part of the body. In simpler terms, this invention, through the establishment of its own standard human body model, uses parameters corresponding to 20 body shapes and 17θ bones to generate a 3D human body model that is closer to the Asian human body type than the Max Planck SMPL model, and offers better independent operation and control. Human body shapes vary greatly, and many people's thigh and calf proportions do not meet a certain precise ratio. Our model can control and adjust the length of the thighs and calves separately by controlling the input parameters, achieving precise determination of leg proportions. In fact, only such a human body model can perfectly match our entire training method, because our training method requires that each part of the human body model be adjustable as individually as possible. This fully leverages the advantages of the training method of this invention, achieving a perfect fit between the human body model and the human body in the photograph in all visible parts.

[0035] 5. Simple User Operation. This invention provides a method for obtaining accurate 3D human body model parameters by analyzing full-body photographs using deep neural networks. It can quickly create a human body model using only a single ordinary photograph, perfectly adapting to the characteristics and trends of the internet age—it's simple and fast. Users don't need any preparation; uploading a photo is all they need to do. Applying this invention to entertainment mini-programs or online shopping scenarios will greatly enhance user experience and engagement. The 3D model, which corresponds to the true form of the human body, can be obtained without a depth-sensing camera or multiple cameras, providing broad application scenarios for various industries such as clothing and health. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a complete process flowchart for one embodiment;

[0038] Figure 2 A flowchart of the model parameter acquisition module for one embodiment;

[0039] Figure 3 A schematic diagram illustrating the human body model correction process in one embodiment;

[0040] Figure 4 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0041] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the invention.

[0042] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0043] The method for processing human images provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] like Figure 1As shown, this embodiment of the invention provides a training method for a three-dimensional human body model neural network. By aligning and adjusting the generated human body model in detail, the parameter accuracy of the deep learning neural network is greatly improved, the convergence speed is significantly accelerated, and the consistency and fidelity of the generated three-dimensional human body model can be greatly improved.

[0045] First, this invention discloses a method for training a neural network for a human body model, comprising:

[0046] 1) Obtain a two-dimensional image of the target human body;

[0047] 2) Obtain the parameters of the target human body model using a neural network, and then obtain a 3D target human body model mesh based on the parameters;

[0048] 3) Import the 2D image and target human body model into the 3D modeling software;

[0049] 4) Use 3D modeling software to make consistency adjustments between the 2D image and the target human body model to obtain a corrected new target human body model;

[0050] 5) Substitute the new target human body model as the output into the neural network for calculation;

[0051] 6) Use the calculation results to adjust the model parameters and complete the training of the neural network.

[0052] The training method also includes: constructing a three-dimensional standard human body model by combining a mathematical model; obtaining three-dimensional human body parameters of the target human body model through a neural network, including three-dimensional human body posture parameters and three-dimensional human body shape parameters; corresponding the obtained three-dimensional human body posture and body shape parameters with several basis and skeletal parameters of the three-dimensional standard human body model; and inputting the obtained several sets of basis and skeletal parameters into the standard three-dimensional standard human body model for fitting.

[0053] The training method generally includes three steps: First, obtain a 3D target human body model with the same posture and body shape as the target human body; second, use 3D modeling software to adjust the consistency between the 2D image and the target human body model; and third, use the corrected human body model to complete the training of the neural network.

[0054] The first part involves processing the acquired human images to obtain the parameter information needed to generate the human model. Previously, the selection of these skeletal key points was usually done manually, but this method is inefficient and unsuitable for the fast-paced demands of the internet age. Therefore, with the widespread use of neural networks, using deep learning-based neural networks to replace manual key point selection has become a trend. However, how to efficiently utilize neural networks is a problem that requires further research. Overall, we adopted a two-level neural network plus data "refinement" approach to construct our parameter acquisition system. For example... Figure 2 As shown, we use a deep learning neural network to generate these parameters, which mainly includes the following sub-steps: 1) Obtain a two-dimensional image of the target human body; 2) Process the image to obtain a two-dimensional human body contour image; 3) Substitute the two-dimensional human body contour image into the first deep learning neural network to perform joint point regression; 4) Obtain the joint point map of the target human body; obtain semantic segmentation maps of various parts of the human body; body key points; body skeleton points; 5) Substitute the generated joint point map, semantic segmentation map, body skeleton points and key point information of the target human body into the second deep learning neural network to perform regression of human posture and body shape parameters; 6) Obtain the output three-dimensional human body parameters, including three-dimensional human body action posture parameters and three-dimensional human body shape parameters.

[0055] The acquisition of a two-dimensional human contour image utilizes an object detection algorithm, which is a target region fast generation network based on a convolutional neural network. Before inputting the two-dimensional human image into the first neural network model, a process of training the neural network is included. The training samples consist of standard two-dimensional human images with annotated original joint point positions, which are manually annotated with high accuracy on the two-dimensional human images. Here, the target image is first acquired, and the object detection algorithm is used to detect human bodies in the target image. Human body detection does not involve using measuring instruments to detect real human bodies; in this invention, it actually refers to any given image, typically a two-dimensional photograph containing sufficient information, such as a face, limbs, and body, all of which must be included in the image. Then, a certain strategy is used to search the given image to determine whether it contains a human body. If the given image contains a human body, parameters such as the human body's position and size are provided. In this embodiment, before obtaining the key points of the human body in the target image, it is necessary to perform human body detection on the target image to obtain the human body bounding boxes that mark the location of the human body in the target image. Since the image we input can be any image, it is inevitable that there will be some non-human body backgrounds, such as tables, chairs, trees, cars, buildings, etc. These useless backgrounds need to be removed by some mature algorithms.

[0056] Simultaneously, we also need to perform semantic segmentation, keypoint detection, skeleton detection, and edge detection. Collecting this 1D point information and 2D surface information lays a solid foundation for generating a 3D human body model. A first-level neural network is used to generate the human body's keypoint map. Optionally, an object detection algorithm can quickly generate a network for target regions based on a convolutional neural network. This first neural network requires extensive data training. Manual annotation of keypoints on photos collected from the network is performed and then input into the neural network for training. After deep learning, the neural network can essentially obtain a keypoint map with the same accuracy and effect as manually annotated keypoints immediately upon inputting a photo, while being tens or even hundreds of times more efficient. Human body keypoints are typically referred to as critical points on the human body.

[0057] In this invention, obtaining the joint positions of the human body in a photograph is only the first step, acquiring 1D point information. The next step is to generate 2D surface information from this 1D point information. These tasks can be accomplished using neural network models and mature algorithms already in use. This invention, by redesigning the process and timing of neural network model intervention and rationally designing various conditions and parameters, makes parameter generation more efficient and reduces the degree of human intervention. This makes it highly suitable for internet application scenarios, such as virtual dress-up programs, where users can obtain the dress-up results almost instantly without waiting, playing a crucial role in enhancing the program's appeal to users.

[0058] After obtaining the relevant 1D point information and 2D surface information, these parameters or results, along with the target human body's joint map, semantic segmentation map, body skeleton points, and keypoint information, can be used as inputs to regress human posture and body shape parameters using a second deep learning neural network. After regression calculation by the second neural network, several sets of 3D human body parameters can be immediately output, including 3D human motion posture parameters and 3D human body shape parameters. Preferably, the loss function of the neural network is designed based on a 3D standard human body model (basic mannequin), a predicted 3D human body model, a standard 2D human body image with annotated original joint point positions, and a standard 2D human body image including predicted joint point positions.

[0059] In this section, we will also pre-design and model some basic mannequins. The main task is to construct a 3D standard human body model, i.e., a basic mannequin, by combining mathematical models. The Max Planck Institute's SMPL human body model, through the study of continuous animation of parameter changes, clearly shows that every continuous change in a parameter controlling human morphology will cause a chain reaction of local or even overall changes in the human body model. To represent the movement of human muscle tissue, a linear change in each parameter of the SMPL human body model will cause a large-area mesh change. To illustrate, for example, when adjusting the parameter β1, the model will directly interpret the change in β1 as a change in the entire body. You might only want to adjust the waist proportion, but the model will forcibly adjust the legs, chest, and even the size of the hands as well. While this working mode can greatly simplify the workflow and improve efficiency, it is indeed very inconvenient for projects that prioritize modeling effects. Because the SMPL human body model is ultimately a model trained using Western anthropometric photographs and measurements, conforming to Western body types, its shape variations generally follow the typical curves of Westerners. Applying it to modeling Asian bodies will lead to many problems, such as issues with arm-to-leg ratios, waist-to-body ratios, neck proportions, and leg and arm lengths. Our research shows significant discrepancies in these aspects. Blindly applying the SMPL human body model will not achieve the desired final result.

[0060] To address this, we employed a self-made human body model to enhance the effect. The core of this approach is the creation of a custom-built human blend body model base to achieve precise and independent manipulation of the human body. The standard 3D human body model (basic mannequin) consists of 20 body shape base parameters and 170 bone parameters. These bases constitute the entire human body model, with each base individually controlled by parameters, without interfering with others. Independent manipulation means that each base is manipulated independently, such as the waist, legs, hands, and head. Each bone can also have its length adjusted independently, without any inter-body movement. This allows for more precise adjustments to the human body model. Our existing model embodies a mathematical correspondence, better representing local shape changes in the human body model. Furthermore, these changes are based on mathematical principles, with no interaction between parameters; arms and legs remain completely independent.

[0061] It is important to emphasize that our human body model is not obtained through training. The parameters have a mathematically based correspondence, meaning that our different sets of parameters are independent and not interconnected. Therefore, our model is more interpretable during transformations and can better represent shape changes in a specific part of the body. In simpler terms, this invention, through the establishment of its own standard human body model, uses parameters corresponding to 20 body shapes and 170 bones to generate a 3D human body model that is closer to the Asian human body type than the Max Planck SMPL model, and offers better independent operation and control. Our model can control and adjust the length of the thighs and calves separately by controlling the input parameters, achieving precise determination of leg proportions. In fact, only such a human body model can perfectly match our entire training method, because our training method requires that each part of the human body model be adjustable as individually as possible. This fully leverages the advantages of the training method of this invention, achieving a perfect fit between the human body model and the human body in the photograph in all visible parts.

[0062] The first part also includes fitting the parameters of the human body model to generate the target human body model. For example... Figure 2 As shown, the process includes the following sub-steps: First, the obtained 3D human posture and body shape parameters are correlated with several basis and skeletal parameters of a 3D standard human model. Second, the obtained sets of basis and skeletal parameters are input into the standard 3D human model for fitting. The 3D human model has a mathematical weight relationship between skeletal points and the model mesh; the determination of skeletal points can be associated with the determination of the target human posture in the human model. In this part, using the two types of parameters generated in the previous part, a pre-designed human model can be substituted to construct the 3D human model. These two types of parameters have similar names to the parameters of the Max Planck Institute's SMPL human model, but their actual content differs significantly. This is because their foundations are different. That is, this invention uses a self-made 3D standard human model (basic human form), while the Max Planck Institute's SMPL model uses a standard human model generated through big data training. The generation and calculation methods of the two models are different. Although both ultimately result in a generated 3D human model, their connotations differ considerably. After this step, a preliminary 3D human model is obtained, including a mesh of the human model containing skeletal positions and length information.

[0063] In the second part, we complete the revision of the human body model.

[0064] The human body model output from our designed neural network model is actually roughly the same in posture as the human body in the original photograph; this is generally referred to as the target human body model. Because the skeletal joint diagrams have undergone continuous training and manual annotation, they already possess a certain degree of consistency. Extreme cases rarely occur where the arms in the photograph are raised, while the arms in the human body model are lowered. Generally, the difference between the target human body model and the actual human posture and body shape in the photograph we judge is not very significant. However, previously, no one further processed these imperfect human body models. The usual practice was to continue adjusting the parameters and weight settings of the neural network through other methods, continuously outputting a parameter-corrected human body model. If the posture became closer, it was considered that the parameter adjustment was relatively successful. However, in terms of efficiency, this adjustment method is highly random, with generally mediocre results and efficiency.

[0065] The training approach of this invention involves manually intervening and adjusting the imperfect human body model generated by the neural network to achieve a state highly consistent with the human posture in the photograph. This corrected model is then used as the human body model that should be generated from the aforementioned photograph, and substituted back into the neural network model for calculation to determine the parameters the neural network should possess to output the correct answer. Compared to various existing methods for obtaining realistic human body model data, the method of this invention is simple and effective, requiring only a single photograph to complete the entire correction and training process, greatly simplifying data acquisition time and reducing equipment costs. Furthermore, it should be emphasized that the optimal correction effect is achieved precisely because we use a self-made standard human body model that can independently control each part. Using the Max Planck Institute's SMPL model would not achieve the optimal effect of independently controlling the model to fit the photograph.

[0066] See Figure 3In this invention, to obtain a human body model with a "perfect" posture, the first step is to import the generated model into mainstream 3D modeling software, such as Maya. Secondly, the original 2D photograph is also imported into the modeling software, displaying both the photograph and the human body model within the same task window. Thirdly, the skeleton and joints of the human body model are adjusted in a separate task window within Maya. The artist manipulates the joints and bones bound to the human body model, referencing areas that differ slightly from the original photograph, to "align" the posture and body shape with the photograph. For example, if the arm in the photograph is raised at a 30-degree angle to the left front, and the generated model is at a 25-degree angle to the left front, a manual adjustment of 5 degrees is sufficient. More complex situations arise when the arm's direction, arm thickness, arm length, and hand rotation direction are inconsistent. In these cases, the artist needs to make individual adjustments. Adjustments can be made using common methods in Maya, such as dragging the entire joint using the IK method to directly adjust the XYZ coordinates and Euler angles. In practice, manual intervention in this step significantly improves model accuracy and overall efficiency. While some software achieves this "fitting" process, it primarily relies on image processing techniques, which can lead to a substantial decrease in accuracy, resulting in less efficient and effective retraining. Although both manual and automated adjustments aim for the human model to completely "cover" the human body in the original photograph, we prefer manual adjustments. However, it's important to emphasize that this doesn't significantly impact the training method claimed in this invention. Regardless of the method used, the goal is to generate a human model that closely resembles the posture and body shape of the human in the original image. This model correction step greatly improves training accuracy, allowing for more precise adjustments to various parameters to generate a target human model that more closely approximates the real human body in the photograph.

[0067] After adjustments, we added an acceptance and refining process. We rendered the three-view drawing of the human model in Maya and compared it to the photograph. If they were highly consistent, the result could be used to train the neural network. If inconsistencies remained, we fine-tuned those areas in the software. While aligning the model and photograph using the human eye within Maya's virtual 3D space is relatively accurate, slight errors are inevitable. Generating a 2D image of the model allows for careful observation of inconsistencies between the photograph and the model in 2D space from a static perspective. This makes it easier to spot inconsistencies that are difficult to detect in 3D space, such as the shape of elbows, backward-bending limbs, and forward or backward leaning bodies. This further assists in refining the model to a more perfect form.

[0068] In Part Four, we re-input the revised human model as the result into the neural network for training, so that the neural network can determine the most accurate match between the input photo and the output model. There are many methods for training neural networks, but the basic process and ideas are similar; simply put, it's about teaching the neural network what is right and what is wrong. One of the commonly used deep learning neural network training methods usually includes: (1) data preprocessing; (2) inputting data into the neural network (each neuron first inputs values, weighted sums, and then inputs the activation function as the output value of the neuron) forward propagation to obtain a score or result; (3) inputting the "score" or "result" into the error function loss function (regularization penalty to prevent overfitting), comparing it with the expected value to obtain the error, and summing multiple errors to judge the degree of recognition (the smaller the loss value, the better); (4) determining the gradient vector through backpropagation (backward differentiation, the error function and each activation function in the neural network are required, the ultimate goal is to minimize the error); (5) finally adjusting each weight through the gradient vector to adjust the trend of the error tending to 0 or convergence towards the "score" or "result"; (6) repeating the above process until the set number of times or the average value of the loss no longer decreases (the lowest point); (7) training is completed.

[0069] As we can see, most neural network training processes are inherently based on large amounts of data. Initially, the neural network model performs poorly in many scenarios, so a large number of bad cases need to be labeled. These labeled bad cases are then added to the training set, allowing the neural network to learn what the true values ​​of these bad cases should be. Once the network learns this, it can accurately predict similar scenes. However, this training method is not very efficient. Therefore, the training process is essentially iterative. If the model can be trained to identify what is correct and good using results that are almost identical to the standard answer, fewer bad cases will appear later, the convergence speed will increase rapidly, and the model performance will improve.

[0070] Combination Figures 1 to 3 The described method flow is a training method for generating a three-dimensional human body model neural network according to an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the hardware structure 300 of a device for processing human body images according to an embodiment of the invention.

[0071] The present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the training method and steps described above.

[0072] And an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the training methods and steps described above when executing the programs stored in the memory.

[0073] like Figure 4 As shown, the device 300 for implementing the neural network training method in this embodiment includes: a processor 301, a memory 302, a communication interface 303, and a bus 310, wherein the processor 301, the memory 302, and the communication interface 303 are connected through the bus 310 and complete communication between them.

[0074] Specifically, the processor 301 may include a central processing unit (CPU), an ASIC, or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0075] Memory 302 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 302 may include an HDD, floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where suitable, memory 302 may include removable or non-removable (or fixed) media. Where suitable, memory 302 may be internal or external to the human image processing device 300. In a particular embodiment, memory 302 is a non-volatile solid-state memory. In a particular embodiment, memory 302 includes read-only memory (ROM). Where suitable, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0076] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of the present invention.

[0077] Bus 310 includes hardware, software, or both, that couples components of device 300 for processing human images together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 310 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0078] In other words, Figure 4 The illustrated device 300 for processing human images can be implemented as including: a processor 301, a memory 302, a communication interface 303, and a bus 310. The processor 301, memory 302, and communication interface 303 are connected via the bus 310 and communicate with each other. The memory 302 stores program code; the processor 301 reads the executable program code stored in the memory 302 to run a program corresponding to the executable program code, thereby executing the neural network training method in any embodiment of the present invention, and thus achieving a combination of... Figures 1 to 3 Describes the neural network training method.

[0079] This invention also provides a computer storage medium storing computer program instructions; when these computer program instructions are executed by a processor, they implement the neural network training method provided in this invention.

[0080] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0081] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0082] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0083] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A method for training a neural network for a human body model, characterized in that, The method comprises: 1) obtaining a two-dimensional image of a target human body; 2) constructing a three-dimensional standard human body model in combination with a mathematical model, the three-dimensional standard human body model being composed of a plurality of body base parameters and a plurality of bone parameters, the plurality of body bases constituting the entire human body model, each body base being individually controlled and changed by the base parameter, and the three-dimensional standard human body model having a mathematical weight relationship between the bone points and the model grid; obtaining three-dimensional human body parameters of the target human body model by using a neural network, including three-dimensional human body action posture parameters and three-dimensional human body size parameters, the three-dimensional human body action posture and three-dimensional human body size parameters corresponding to the plurality of body bases and bone parameters of the three-dimensional standard human body model, inputting the plurality of sets of base and bone parameters obtained into the three-dimensional standard human body model for fitting to obtain a three-dimensional target human body model grid; 3) importing the two-dimensional image and the target human body model into a three-dimensional modeling software; 4) adjusting the two-dimensional image and the target human body model for consistency by using the three-dimensional modeling software to obtain a corrected new target human body model, including: After importing the two-dimensional image and the target human body model into the three-dimensional modeling software, adjusting the human body model bone parameters to realize posture consistency and adjusting the body base parameters to realize size consistency to obtain the corrected new target human body model; rendering three views of the human body model in the three-dimensional modeling software, comparing the three views with the two-dimensional image, if consistent, taking the three views as the output result to train the neural network, if inconsistent, fine-tuning the inconsistent parts again in the three-dimensional modeling software to obtain the corrected new target human body model; 5) inputting the new target human body model as the output result into the neural network, calculating the predicted value by forward propagation, and calculating the parameter error by back propagation algorithm; 6) iteratively adjusting the model parameters by using the calculation result to complete the training of the neural network, including: (1) inputting the preprocessed data into the neural network for forward propagation to obtain a score or a result; (2) inputting the score or the result into an error function to identify the degree of error; (3) determining a gradient vector by back propagation; (4) adjusting the weight value by the gradient vector to adjust the score or the result to the trend of tending to 0 or convergence; (5) repeating the above iteration until the set number of times or the average value of the loss error no longer decreases; (6) training is completed.

2. The method of claim 1, wherein, The training method further comprises: obtaining three-dimensional human body parameters of the target human body model by using the neural network.

3. The method of claim 1, wherein, The step of obtaining the parameters of the target human body model further comprises: 1) obtaining a two-dimensional image of a target human body; 2) processing a two-dimensional human body contour image of the target human body; 3) inputting the two-dimensional human body contour image into a first neural network trained by deep learning to regress the joint points; 4) obtaining a joint point map of the target human body; obtaining a semantic segmentation map of each part of the human body; and a body key point. Body skeleton points; 5) the generated target human body joint point graph, semantic segmentation graph, body skeleton points and key point information are substituted into the second neural network subjected to deep learning to regress human body posture and body shape parameters; 6) three-dimensional human body parameters including three-dimensional human body action posture parameters and three-dimensional human body shape parameters are obtained.

4. The method of claim 3, wherein, Before the two-dimensional human body image is input into the first neural network model, a process of training the first neural network is further included, and training samples include standard two-dimensional human body images with original joint point positions labeled by manual high-accuracy labeling on the two-dimensional human body images.

5. The method of claim 4, wherein, The posture consistency between the imported target human body model and the two-dimensional image of the target human body is realized by adjusting the human body model skeleton, and the body shape consistency is realized by adjusting the human body model body base.

6. The method of claim 1, wherein, After obtaining the corrected new target human body model, three views of the human body model are rendered in the software, and supplementary correction and adjustment are performed after comparison with the photo.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-6.

8. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-6.

Images