A driving method, device and storage medium of a mannequin

Abstract

The application discloses a driving method of a human model, which comprises the following steps: acquiring the posture and body shape parameters of a target human model, wherein the three-dimensional human posture and body shape parameters correspond to the skeleton and a plurality of base parameters of a three-dimensional standard human model; establishing a standard human model in an initial posture; inputting the obtained base and skeleton parameters into the standard three-dimensional human model for fitting; obtaining a three-dimensional human model with the same body shape as the target human; completing the driving of the skeleton from the initial posture to a target posture; and obtaining a three-dimensional target human model grid with the same posture as a two-dimensional image of the target human. The fitting of the standard human model and the target human model is optimized, an animation series from the initial posture to the target posture is formed, and the driving process of the skeleton movement is optimized by using a variable-speed frame interpolation method, so that the whole fitting process achieves a good balance in time and effect.

Description

Technical Field

[0001] This invention belongs to the field of human body 3D modeling, and specifically relates to a human body model driving method, especially a method, device and storage medium for driving a standard human body model and a target human body model based on frame interpolation technology. Background Technology

[0002] With the development of internet technology, online shopping has become increasingly popular. Compared to shopping in physical stores, online shopping offers advantages such as a wider variety of goods and greater convenience. However, buying goods online also presents some unresolved problems, the most significant being the inability to physically view the items being purchased. This issue is most pronounced for clothing. Unlike physical stores where customers can try on clothes and see how they look, online clothing stores cannot provide personalized images; they only offer pictures of models trying on clothes, and some online stores don't even offer any pictures of models trying on clothes. Consumers cannot directly assess how well the clothing matches their own body type and appearance, leading to a large number of returns and exchanges.

[0003] To address this issue, businesses have attempted to use virtual try-on technology to provide consumers with a simulated fitting experience. Of course, there are other real-world applications where virtual try-on technology can be used, such as in online games. Therefore, this technology has experienced rapid development.

[0004] Virtual try-on refers to a technology that allows users to view the "fitted" effect in real-time on a terminal screen without actually trying on the clothes they want to see. Existing virtual try-on technologies mainly include 2D virtual try-on and 3D virtual try-on. The former basically involves collecting images of the user and the clothing, then cropping and stitching them together to create an image of the "dressed" person. However, this type of image suffers from poor realism due to its simplistic image processing methods, completely disregarding the user's actual body shape and merely applying the clothing to a photo, failing to meet user needs. The latter typically uses 3D acquisition equipment to collect 3D information about the person and combines it with clothing features for synthesis, or it involves manually inputting body data provided by the user and generating a virtual 3D human model according to certain rules, which is then combined with clothing textures. Overall, this type of 3D virtual try-on requires extensive data collection or 3D data calculation, resulting in high hardware costs and making it difficult to promote among ordinary users.

[0005] With the development of cloud computing, artificial intelligence, and smart terminal processing capabilities, two-dimensional virtual fitting technology has emerged. This technology mainly includes three steps: (1) processing the personal body information provided by the user to obtain a target human body model; (2) processing the clothing information to obtain a clothing model; and (3) merging the human body model and the clothing model together to generate a simulation image of a person wearing the clothing.

[0006] However, due to the accumulation of many uncertainties such as process design, model parameter selection, and neural network training methods, the quality of the final generated clothing images is not as good as that of traditional 3D virtual try-on technology. Among these, the fitting of the human body model is a basic step, and the subsequent dressing process must also be based on the human body model generated earlier. Therefore, if the human body model is not generated accurately, it is easy to produce problems such as a large difference between the human body model and the body shape of the try-on person, loss of skin texture, and loss of body parts, which will affect the final quality of the generated clothing images.

[0007] In the general field of computer vision, there are many starting points for human body modeling, typically including three main categories: omnidirectional scanning of a real human body using 3D scanning equipment; 3D reconstruction methods based on multi-view depth photography; and methods combining given images with a human body model to achieve 3D reconstruction. Among these, omnidirectional scanning of a real human body using 3D scanning equipment yields the most information and is the most accurate. However, such equipment is usually expensive and requires a high degree of cooperation from the human model. The entire processing process places very high demands on the processing equipment, so it is generally used in specialized fields. Secondly, multi-view 3D reconstruction methods require providing overlapping images of the reconstructed human body from multiple perspectives and establishing spatial transformation relationships between the images. Using multiple cameras to capture multiple images and then stitching them together to create a 3D model simplifies the operation somewhat, but the computational complexity remains high. Furthermore, in most cases, only people present at the scene can obtain multi-angle images. The model obtained by stitching together textures from multi-angle depth camera photography lacks body-scale data and cannot provide a foundation for 3D perception. Secondly, the method combining a single image with a human body model only requires one image. The intelligent generation method of 3D human body feature curves based on neural networks, through neural network training, obtains weights and thresholds that can describe curves of parts of the human body such as the neck, chest, waist, and hips. Then, based on the dimensional parameters such as the circumference, width, and thickness of the human body cross-section, it can directly generate a 3D human body curve that matches the real human body shape, thus obtaining a predicted human body model. However, this method suffers from limited input information and still requires a significant amount of computation, resulting in unsatisfactory final model performance.

[0008] Given the characteristics of internet technology and the network environment, directly outputting the final human body model from a single image is undoubtedly the preferred method, offering the greatest convenience. Users don't need to be physically present; a single photograph is sufficient to complete the entire clothing-changing process. The ensuing question is: if the resulting photograph can be guaranteed to be roughly equivalent to a realistic 3D simulation of clothing changing, it will become the mainstream approach. Therefore, the crucial question becomes: how to obtain a human body model that most closely resembles the real human form from a single photograph?

[0009] In the existing technology, there are usually several types of methods for constructing human body models: (1) Regression-based methods, which reconstruct the human body model represented by voxels through convolutional neural networks. The algorithm first estimates the position of the main joints of the human body based on the input image, and then estimates the position of the key points. In a given voxel grid of a specified size, the shape of the reconstructed human body is described by the shape of the entire voxel occupied by the voxel. (2) Human body reconstruction based on a single image. This method estimates the three-dimensional shape and posture of the human body at the same time. This method first roughly marks the simple human skeleton key points on the image, and then performs initial matching and fitting of the human body model based on these coarse key points to obtain the approximate shape of the human body. (3) Use 23 bone nodes to represent the human skeleton, and then use the rotation of each bone node to represent the posture of the entire human body. At the same time, use 6890 vertex positions to express the human body shape. In the fitting process, given the bone node positions, the parameters of shape and posture are fitted at the same time to perform three-dimensional human body reconstruction. Alternatively, a CNN model is used to predict the key points on the image, and then an SMPL model is used for fitting to obtain the initial human body model. Next, the fitted shape parameters are used to regress a human joint bounding box, with each joint corresponding to a bounding box, represented by its axis length and radius. Finally, the initial model and the regressed bounding boxes are combined to obtain a 3D human reconstruction. The above method suffers from slow modeling speed, insufficient modeling accuracy, and a strong dependence on the created body and pose database for reconstruction results.

[0010] Existing technology 1 discloses a human body modeling method based on body measurement data, such as Figure 1 As shown, the method includes: acquiring body measurement data; performing linear regression on a pre-created human body model using a pre-trained prediction model based on the body measurement data to obtain a predicted human body model, wherein the pre-created human body model includes multiple predefined sets of labeled feature points and corresponding standard shape bases, and the body measurement data includes measurement data corresponding to each set of labeled feature points; and obtaining a target human body model based on the predicted human body model, wherein the target human body model includes measurement data, a target shape base, and a target shape coefficient. However, this method has very high requirements for body measurement data, including body length and circumference data, such as height, arm length, shoulder width, leg length, calf length, thigh length, foot length, head circumference, chest circumference, waist circumference, and thigh circumference, etc., which not only requires measurement but also calculation. While it does save computation, the user experience is very poor, and the program is cumbersome. Furthermore, the training method of the SMPL model is referenced in the training of the human body model.

[0011] The SMPL model is a parametric human body model, a human body modeling method proposed by the Max Planck Institute. This method can perform arbitrary human body modeling and animation-driven operations. The biggest difference between this method and traditional LBS (Location-Based Services) lies in its proposed method of human posture image surface morphology. This method can simulate the bulges and depressions of muscles during limb movement. Therefore, it can avoid surface distortion of the human body during movement and accurately depict the morphology of muscle stretching and contraction movements. In this method, β and θ are the input parameters. β represents 10 parameters related to individual body height, weight, head-to-body ratio, etc., while θ represents 75 parameters representing the overall human body posture and the relative angles of 24 joints. However, the core of this model generation method is the accumulation of a large amount of training data to obtain the relationship between body shape and shape basis. However, due to the strong interrelationships between these relationships, each shape basis cannot be independently controlled, making decoupling operations difficult. For example, there is also a certain correlation between arms and legs; when the arm moves, the leg should theoretically move as well. It is difficult to achieve improvements for different body shapes using the SMPL model.

[0012] Existing technology 2 discloses a 3D human body modeling method based on a single photograph, including: acquiring a photograph, analyzing the photograph, marking key points of the human body in the photograph, and calculating the spatial coordinates of the key points; obtaining the distance between the skeletal points in a pre-created standard human body model and the key points in the photograph, aligning the skeletal points with the key points to generate a basic human body model; acquiring the basic texture map in the pre-created standard human body model, performing difference calculation between the basic texture map and the skin texture of the face in the photograph, and then using the edge channel to fuse them to generate basic texture data; and generating a 3D human body model based on the basic human body model and the basic texture data. This method achieves 3D human body modeling from a single photograph, and the model has a skeletal and muscular system for support, enabling expressions and movements. However, this method matches the distance between key points in the user's photograph and key points on a standard mannequin, then adjusts the distance to achieve the target human body's posture. Furthermore, it requires subsequent difference calculation and fusion between the basic texture map and the skin texture in the photograph to obtain the final human body model. While this method is simple and computationally inexpensive, the accuracy and realism of the generated human body model are not very high.

[0013] Existing technology 3 discloses a method for generating a three-dimensional human body model, including: acquiring a two-dimensional human body image; inputting the two-dimensional human body image into a three-dimensional human body parameter model to obtain three-dimensional human body parameters corresponding to the two-dimensional human body image; inputting the training samples into a neural network for training to obtain a three-dimensional human body parameter model, including: inputting the standard two-dimensional human body image in the training samples into the neural network to obtain predicted three-dimensional human body parameters corresponding to the standard two-dimensional human body image; adjusting a three-dimensional flexible deformable model according to the predicted three-dimensional human body parameters to obtain a predicted three-dimensional human body model; and obtaining the predicted joint point positions in the standard two-dimensional human body image through back mapping based on the joint point positions in the predicted three-dimensional human body model. This modeling method utilizes model judgment and the parameters output by the neural network, which only relate to node parameters. It then uses the mature body shape of the SMPL model to perform detailed adjustments to match the target human body posture. Although this reduces the computational load, due to the limited input parameters and the fact that adjustments can only be made based on the SMPL prediction model, it is difficult to output a human body model that is highly consistent with the target human body posture.

[0014] Therefore, to align with the development trends of the internet industry, in the niche field of virtual try-on, the three fundamental goals that are constantly pursued are minimal input information, minimal computational load, and optimal results. We need to find an optimal balance among these three, providing a human body modeling method that allows for simple input, computational loads not exceeding the capacity of the terminal device, and results approaching those of professional equipment. Summary of the Invention

[0015] In view of the above problems, the present invention provides a driving method, device and storage medium for a human body model that overcomes the above problems.

[0016] This invention provides a method for driving a human body model, the method comprising: establishing a standard human body model in an initial posture; acquiring the posture and body shape parameters of a target human body model, wherein the three-dimensional human body posture and body shape parameters correspond to the skeleton and several basis parameters of the three-dimensional standard human body model; inputting the acquired several sets of basis and skeleton parameters into the standard three-dimensional human body model for fitting; completing the movement of the skeleton from the initial posture to the target posture; and obtaining a three-dimensional target human body model mesh with the same posture as the two-dimensional image of the target human body.

[0017] Preferably, the driving process further includes obtaining the position coordinates of the initial pose and the target pose; the initial pose parameters are determined by the initialization parameters of the standard mannequin model, and the skeletal information of the target pose is obtained by regression prediction from a neural network model. The driving process also includes generating an animation sequence from the initial pose to the target pose. After obtaining the initial state of the skeletal information and the target pose state parameters, a time series of skeletal information from the initial pose to the target pose is formed by frame interpolation using linear interpolation or nearest neighbor interpolation. During the driving process, the parent bone nodes are driven first, followed by the child bone nodes. In the process of generating the animation sequence, a mesh frame interpolation method is used. After driving the skeletal movement in each frame, the vertex and face information of the current state of the mannequin model is calculated using the weight parameters of the standard mannequin model, the current mesh state of the mannequin model is updated, and the information is recorded and saved.

[0018] Preferably, the frame interpolation speed adopts a non-uniform frame interpolation rate, which is set to be slow at the position from the initial point and the target point, and fast in the middle of the movement process. That is, the single frame movement amplitude is small during the initial action and the end action, and the movement amplitude is large in the middle of the movement process.

[0019] Preferably, during the generation of the animation sequence, the driving speed is reduced when the frame interpolation is about to end, and the frame is paused for several frames when moving to the final target posture to obtain the entire animation sequence.

[0020] Preferably, the standard human body model is a three-dimensional standard human body model constructed by combining a mathematical model; the three-dimensional standard human body model has a mathematical weight relationship between skeletal points and model mesh, and the determination of skeletal points can be associated with the human body model that determines the target human body posture; the three-dimensional standard human body model is defined by several body base parameters and several skeletal parameters, and the several body bases constitute the entire human body model mesh, and each body base is individually controlled by the base parameters and does not affect each other.

[0021] Preferably, the step of obtaining the parameters of the target human body model further includes: 1) obtaining a two-dimensional image of the target human body; 2) processing to obtain a two-dimensional human body contour image of the target human body; 3) substituting the two-dimensional human body contour image into a first neural network after deep learning to perform joint point regression; 4) obtaining a joint point map of the target human body; obtaining semantic segmentation maps of various parts of the human body; body key points; body skeletal points; 5) substituting the generated joint point map, semantic segmentation map, body skeletal points and key point information of the target human body into a second neural network after deep learning to perform regression of human posture and body shape parameters; 6) obtaining the output three-dimensional human body parameters, including three-dimensional human body action posture parameters and three-dimensional human body shape parameters.

[0022] Preferably, the method further includes acquiring the two-dimensional human contour image using a target detection algorithm, wherein the target detection algorithm is a target region fast generation network based on a convolutional neural network; before inputting the two-dimensional human image into the first neural network model, the method further includes a process of training the neural network, wherein the training samples include standard two-dimensional human images with original joint point positions marked, wherein the original joint point positions are marked with high accuracy by humans on the two-dimensional human images.

[0023] A computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the steps described above.

[0024] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements any of the steps described above.

[0025] The beneficial effects of this invention are:

[0026] 1. Fast model fitting speed and high posture restoration accuracy after fitting. To realistically fit the posture of the target human body, we adopted an optimized frame interpolation method during the transition from the initial pose to the target pose. Compared with traditional frame interpolation, the skeletal information of our target pose is obtained by model regression prediction. Simultaneously, an animation sequence from the initial pose to the target pose is generated. Linear interpolation, nearest neighbor interpolation, and other frame interpolation methods are used to form a time series of skeletal information from the initial pose to the target pose. During the generation of the animation sequence, mesh frame interpolation is used. The frame interpolation speed is set to be slow at the beginning and end relative to the initial and target points, and fast during the intermediate movement. Crucially, several frames are paused when the model reaches the final target pose, providing an effective buffer after high-speed movement before coming to a stop, resulting in a complete animation sequence and higher posture fitting accuracy for the human body model. This approach is closer to the laws of motion in the real physical world than uniform frame interpolation, producing better simulations of clothing and human posture, and significantly reducing processing time.

[0027] 2. Simple User Operation. This invention provides a method for obtaining accurate 3D human body model parameters by analyzing full-body photographs using deep neural networks. It can quickly create a human body model using only a single ordinary photograph, perfectly adapting to the characteristics and trends of the internet age—it's simple and fast. Users don't need any preparation; uploading a photo is all they need to do. Applying this invention to entertainment mini-programs or online shopping scenarios will greatly enhance user experience and engagement. The 3D model, which corresponds to the true form of the human body, can be obtained without a depth camera or multiple cameras, providing broad application scenarios for various industries such as clothing and health.

[0028] 3. High-frequency and creative use of hierarchical deep neural networks. This invention fully leverages the advantages of deep learning networks, enabling high-precision reconstruction of human posture and body shape in various complex scenarios. Different neural networks are used for different purposes, utilizing neural network models with varying input conditions and training methods to achieve accurate contour separation, semantic segmentation, and keypoint and joint determination of the human body against complex backgrounds. This eliminates the influence of loose clothing and hairstyles, achieving the closest possible approximation to the real human body shape and form. While existing technologies also use neural network models, differences in input conditions, input parameters, and training methods lead to significant variations in the functions and effects of these models.

[0029] 4. The neural network model is set up more scientifically and purposefully. Some existing image processing methods overemphasize simply outputting models directly without spending time refining the model's details. They rely solely on training with massive amounts of image data to map 2D images to 3D body models. While this is highly efficient, the processing flow is too simplistic. Relying entirely on neural network models to generate 3D human models results in unsatisfactory consistency and quality in body proportions and details. Furthermore, it offers no help for subsequent processing and may even become a major obstacle for later stages. In contrast, our preceding neural network takes human contours, semantic segmentation, keypoints, and joints as inputs, generating model parameters from multiple perspectives. The subsequent neural network outputs parameters in two categories: pose and shape, allowing for separate control of movement and body shape. Combined with our reference model, it can accurately replicate the pose and body shape of the human model.

[0030] 5. Precise and Controllable Human Body Model. Currently popular single-image-based human body reconstruction methods mainly fall into two categories: reconstructing parametric human body models. The most commonly used parametric model is the Max Planck Institute's SMPL model, which contains two sets of 72 parameters describing human posture and body shape. For single-image reconstruction, the SMPL parameters are first estimated from the image, and then optimized by minimizing the projection distance between the 3D joints and the 2D planar joints, thus obtaining the human body. However, the SMPL model is mainly trained through a large number of human body model instances using deep learning. The relationship between body shape and shape basis is a holistic one, which is very difficult to decouple. It is impossible to control the desired body parts at will, resulting in a model that cannot achieve a high degree of consistency with the real human posture and body shape. In addition, if it is further applied to the subsequent dressing process, it will also have limited ability to represent the geometric details of the human body surface and cannot reconstruct the detailed texture of clothing on the human body surface well. However, our human body model is not obtained through training. The parameters have a correspondence based on mathematical principles. In other words, our different sets of parameters are independent of each other and have no interrelationship. Therefore, our model is more interpretable during transformation and can better represent the shape changes of a certain part of the body. In layman's terms, everyone's body shape is different, and many people's thigh and calf ratios do not meet a certain precise proportion. Our model can control the input parameters to achieve separate control and length adjustment of the thigh and calf, thus achieving a precise determination of the leg proportions.

[0031] 6. More suitable for Asian body types. Human body modeling typically involves designing standard human body models, also known as standard human body models or basic mannequins. Using a self-built standard human body model allows for the manipulation of the human body, specifically controlling the process from the initial posture to the target posture. This is fundamental for the subsequent clothing to adapt to changes in the body's posture. Only when the human body accurately reaches the target posture can the specific process of the clothing following the body to that posture be calculated. In this process, we abandoned the basic human body models trained on European human body data using the Max Planck SMPL model, and instead built our own standard human body model (standard mannequin) that is more suitable for Asian body types. This mannequin can ideally include parameters for 170 bones and 20 shape bases, greatly enriching the details of the human body model and surpassing the SMPL model in terms of detail expression. Furthermore, combined with the aforementioned feature of independent control for each base, each part of the mannequin can be independently and precisely controlled and modified according to needs, achieving a more aesthetically pleasing effect for each human body model. In addition, we also perform post-production manual adjustments to the mannequin's features, such as the number of vertices and faces, a function that other models, such as the SMPL model, cannot accomplish. Besides the height of the mannequin being precisely adjustable, other aspects such as body shape (weight or size), arm length, leg proportions, waist length, and waist circumference can also be precisely controlled, making the mannequin more closely fit the user's body shape.

[0032] This invention optimizes the fitting of standard human body models and target human body models, forming an animation series from the initial pose to the target pose. It also uses variable-speed frame interpolation to optimize the process of driving skeletal movement, achieving a good balance between time and effect in the entire fitting process. Furthermore, using a proprietary standard human body model, it selects a shape base suitable for Asian body types, generating a 3D human body model that is even closer to Asian body types than the Max Planck SMPL model, offering better independent operation and control. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating the human model fitting process in one embodiment;

[0035] Figure 2 A flowchart of the model parameter acquisition module for one embodiment;

[0036] Figure 3A complete modeling process flowchart for one embodiment;

[0037] Figure 4 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0038] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the invention.

[0039] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0040] The method for processing human images provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0041] like Figure 3 As shown, this embodiment of the invention provides a method for driving a human body model. The method includes: establishing a standard human body model in an initial posture; acquiring the posture and body shape parameters of a target human body model, wherein the three-dimensional human posture and body shape parameters correspond to the skeleton and several basis parameters of the three-dimensional standard human body model; inputting the acquired several sets of basis and skeleton parameters into the standard three-dimensional human body model for fitting; obtaining a three-dimensional human body model with the same body shape as the target human body; completing the skeleton movement from the initial posture to the target posture; and obtaining a three-dimensional target human body model mesh with the same posture as the two-dimensional image of the target human body. Through pre-set deep learning and simulated motion, the computational load and complexity of generating a three-dimensional human body model can be greatly reduced, and the realism of the human body model generated from existing 2D images can be achieved far exceeding that of existing 2D image-generated human body models.

[0042] First, the human body modeling method disclosed in this invention generally includes three steps.

[0043] The first part involves processing the acquired human images to obtain the parameter information needed to generate the human model. Previously, the selection of these skeletal key points was usually done manually, but this method is inefficient and unsuitable for the fast-paced demands of the internet age. Therefore, with the widespread use of neural networks, using deep learning-based neural networks to replace manual key point selection has become a trend. However, how to efficiently utilize neural networks is a problem that requires further research. Overall, we adopted a two-level neural network plus data "refinement" approach to construct our parameter acquisition system. For example... Figure 2 As shown, we use a deep learning neural network to generate these parameters, which mainly includes the following sub-steps: 1) Obtain a two-dimensional image of the target human body; 2) Process the image to obtain a two-dimensional human body contour image; 3) Substitute the two-dimensional human body contour image into the first deep learning neural network to perform joint point regression; 4) Obtain the joint point map of the target human body; obtain semantic segmentation maps of various parts of the human body; body key points; body skeleton points; 5) Substitute the generated joint point map, semantic segmentation map, body skeleton points and key point information of the target human body into the second deep learning neural network to perform regression of human posture and body shape parameters; 6) Obtain the output three-dimensional human body parameters, including three-dimensional human body action posture parameters and three-dimensional human body shape parameters.

[0044] The two-dimensional image of the target human body can be a two-dimensional image including a human figure in any pose and with any clothing. The acquisition of the two-dimensional human body contour image utilizes a target detection algorithm, which is a target region fast generation network based on a convolutional neural network.

[0045] Before inputting the two-dimensional human image into the first neural network model, a process of training the neural network is included. The training samples include standard two-dimensional human images with original keypoint positions annotated. These original keypoint positions are manually annotated with high accuracy on the two-dimensional human images. Here, the target image is first acquired, and a target detection algorithm is used to detect human bodies in the target image. Human body detection does not involve using measuring instruments to detect real human bodies. In this invention, it actually refers to any given image, usually a two-dimensional photograph containing sufficient information, such as a face, and requiring that all limbs and the body of the person be included in the image. Then, a certain strategy is used to search the given image to determine whether the given image contains a human body. If the given image contains a human body, parameters such as the position and size of the human body are given. In this embodiment, before obtaining the human body keypoints in the target image, human body detection needs to be performed on the target image to obtain the human body bounding boxes annotating the human body positions in the target image. Because the input image can be any image, there will inevitably be some non-human body backgrounds, such as tables, chairs, trees, cars, buildings, etc. These useless backgrounds need to be removed using some mature algorithms.

[0046] Simultaneously, we also need to perform semantic segmentation, keypoint detection, skeleton detection, and edge detection. Collecting this 1D point information and 2D surface information lays a solid foundation for generating a 3D human body model. A first-level neural network is used to generate the human body's keypoint map. Optionally, an object detection algorithm can quickly generate a network for target regions based on a convolutional neural network. This first neural network requires extensive data training. Manual annotation of keypoints on photos collected from the network is performed and then input into the neural network for training. After deep learning, the neural network can essentially obtain a keypoint map with the same accuracy and effect as manually annotated keypoints immediately upon inputting a photo, while being tens or even hundreds of times more efficient.

[0047] In this invention, obtaining the joint positions of the human body in a photograph is only the first step, acquiring 1D point information. The next step is to generate 2D surface information from this 1D point information. These tasks can be accomplished using neural network models and mature algorithms already in use. This invention, by redesigning the process and timing of neural network model intervention and rationally designing various conditions and parameters, makes parameter generation more efficient and reduces the degree of human intervention. This makes it highly suitable for internet application scenarios, such as virtual dress-up programs, where users can obtain the dress-up results almost instantly without waiting, playing a crucial role in enhancing the program's appeal to users.

[0048] After obtaining the relevant 1D point information and 2D surface information, these parameters or results, along with the joint map, semantic segmentation map, body skeleton points, and / or keypoint information of the target human body, can be used as input to a second deep learning neural network to regress human posture and body shape parameters. After regression calculation by the second neural network, several sets of 3D human body parameters can be immediately output, including 3D human motion posture parameters and 3D human body shape parameters. Preferably, the loss function of the neural network is designed based on a 3D standard human body model (basic mannequin), a predicted 3D human body model, a standard 2D human body image with annotated original joint point positions, and a standard 2D human body image including predicted joint point positions.

[0049] The second part involves pre-designing and modeling some basic mannequins. The main task is to construct a 3D standard human body model, or basic mannequin, by combining mathematical models. The Max Planck Institute's SMPL human body model avoids surface distortion during human movement and accurately depicts the shape of muscle stretching and contraction. In this method, β and θ are input parameters. β represents 10 parameters related to individual height, weight, head-to-body ratio, etc., while θ represents 75 parameters related to the overall human body posture and the relative angles of 24 joints. The β parameter is a shape-blend pose parameter, which can control the shape changes of the human body through 10 incremental templates. Specifically, the changes in human body shape controlled by each parameter can be depicted through animation. By studying the continuous animation of parameter changes, we can clearly see that each continuous change in the parameter controlling the human body shape will cause a chain reaction of local and even overall changes in the human body model. To reflect the movement of human muscle tissue, the linear change of each parameter in the SMPL human body model will cause large-area mesh changes. To put it simply, when adjusting the β1 parameter, the model interprets the change in β1 as a change in the entire body. You might only want to adjust the waistline, but the model will forcibly adjust the legs, chest, and even the size of the hands as well. While this workflow can greatly simplify processes and improve efficiency, it is indeed very inconvenient for projects that prioritize modeling quality. This is because the SMPL human body model is ultimately a model trained on Western anthropometric photographs and measurements, conforming to Western body types. Its shape changes generally follow the typical curves of Westerners. When applied to modeling Asian bodies, many problems arise, such as the proportions of arms and legs, waist, neck, leg length, and arm length. Our research shows significant discrepancies in these aspects. If the SMPL human body model is rigidly applied, the final generated result will not meet our requirements.

[0050] To address this, we employed a self-made human body model to enhance the effect. The core of this approach is the creation of a custom-built human blend body type base to achieve precise and independent manipulation of the human body. The 3D standard human body model has a mathematical weighting relationship between skeletal points and the model mesh. The determination of skeletal points can be correlated with the human body model that defines the target human posture. The 3D standard human body model is defined by several body type parameters and several bone parameters. These body type parameters constitute the entire human body model mesh, and each body type is individually controlled by its own parameters, without affecting others. Preferably, the 3D standard human body model (basic mannequin) consists of 20 body type parameters and 170 bone parameters. Precise manipulation involves increasing the number of controllable parameters. Instead of using the ten β control parameters from the Max Planck Institute, adjustable parameters now include not only the usual body shape (thickness / fatness) but also arm length, leg length, and the thickness / fatness of the waist, hips, and chest. This more than doubles the number of skeletal parameters, significantly enriching the range of adjustable parameters and providing a solid foundation for the refined design of the standard human body model. Independent manipulation means that each component is manipulated individually, such as the waist, legs, hands, and head. Each bone can also have its length adjusted independently, without any inter-component movement. This allows for more precise adjustments to the human body model, preventing it from appearing bulky and unnatural, and ensuring it never reaches the designer's desired shape. Our existing model embodies a mathematical correspondence, essentially redesigning it from both aesthetic and statistical data perspectives. This allows it to generate a model we believe conforms to Asian body types according to our design rules, significantly different from the large-scale training model of the SMPL human body model. Therefore, our parameter transformations are more interpretable, better representing local shape changes in the human body model. Furthermore, these changes are based on mathematical principles, with no inter-parameter influence; arms and legs remain completely independent. In fact, designing so many different parameters is to avoid the shortcomings of large-scale training models, allowing for precise control of the human body model across more dimensions, not just limited to a few indicators like height, thus significantly improving modeling results. Only when the self-built form base is established can setting so many independent control parameters have practical significance. To meet the requirements of a designer, both are indispensable.

[0051] The third part involves fitting the parameters of the human body model to the human body model itself. For example... Figure 1As shown, the process includes the following sub-steps: matching the obtained 3D human pose and body shape parameters with several basis and skeletal parameters of a 3D standard human model; inputting the obtained several sets of basis and skeletal parameters into a standard 3D human parameter model for fitting; obtaining a 3D human model with the same body shape as the target human; completing the movement of the skeleton from the initial pose to the target pose; and obtaining a 3D target human model mesh with the same pose as the 2D image of the target human.

[0052] The 3D human body model has a mathematical weight relationship between skeletal points and model meshes. The determination of skeletal points can be associated with the determination of the target human body posture in the human body model. In this part, using the two types of parameters generated in the previous part, the pre-designed human body model can be substituted to construct the 3D human body model. These two types of parameters are similar in name to the parameters of the Max Planck Institute's human body SMPL model, but the actual content they contain is quite different. This is because their foundations are different. That is, this invention uses a self-made 3D standard human body model (basic mannequin), while the Max Planck Institute's SMPL model uses a standard human body model generated by big data training. The two models are generated and calculated in different ways. Although both are ultimately manifested as generated 3D human body models, their connotations are quite different. After this step, a preliminary 3D human body model is obtained, including the human body model mesh containing skeletal position and length information.

[0053] In this part, the human model also needs to complete the transformation from the initial pose to the target pose. Since our input is only a photograph, the target human pose in the photograph is usually different from the base mannequin. Therefore, in order to fit the target human's pose, the transformation from the initial pose to the target pose must be completed. To more realistically simulate the model's motion, the following steps are also included when fitting several sets of base and skeletal parameters within a standard 3D human parametric model:

[0054] 1) Obtain the position coordinates of the initial pose and the target pose; the initial pose parameters are determined by the initialization parameters of the standard mannequin model, and the skeletal information of the target pose is obtained by regression prediction from the neural network model.

[0055] 2) Generate an animation sequence from the initial pose to the target pose; after obtaining the initial state of the skeleton information and the target pose state parameters, a time sequence of skeleton information from the initial pose to the target pose is formed through frame interpolation methods such as linear interpolation and nearest neighbor interpolation. During the driving process, depending on the number of bones driven per frame, there are two methods: global linear interpolation and driving the parent node first and then the child node. Considering the driving state in the simulated physical world, this patent adopts the latter method, driving the parent bone node first and then the child bone node. This interpolated animation sequence is more closely aligned with the real physical world, resulting in a better simulation effect.

[0056] 3) During the generation of the animation sequence, a mesh frame interpolation method is used; that is, after each frame drives the skeletal movement, the vertex and face information of the human model in the current state is calculated through the weight parameters of the standard human figure, the current human model mesh state is updated and recorded. This is also a key step to ensure the model's fidelity, which can reduce the unwanted deformation and distortion of the human model to an acceptable level.

[0057] 4) The frame interpolation speed is set to be slow at the initial and target points and fast during the middle of the movement. This patent uses a non-uniform frame interpolation rate, that is, the single-frame movement amplitude is small during the initial and final movements, and the movement amplitude is large during the middle of the movement. It simulates the initial state of physical action in the real world with a certain acceleration process, while maintaining a high inter-frame displacement distance during the movement, and reducing the driving speed at the end of the movement.

[0058] 5) Pause for several frames when driving to the final target posture to obtain the entire animation sequence. This approach is closer to the motion laws of the real physical world than uniform frame interpolation. It allows the model to have an effective buffer after high-speed movement and before coming to a stop, thus obtaining the entire complete animation sequence. A region with denser frame interpolation is formed near the final posture, resulting in higher posture fitting accuracy of the human model and better simulation effects.

[0059] 6) Complete the movement of the skeleton from the initial pose to the target pose; at this point, the standard human model has become a human model with a pose and body shape that are basically consistent with the target human model, with a natural pose and no clipping.

[0060] Since we already have the data on the skeleton and the information on the mesh, the skeleton driving becomes easier in this case. We can use the LBS algorithm (skeleton skinning) and the DQS algorithm. Of course, colliders also need to be considered, because the standard mannequin model is in a standard 3D pose. The change from the initial pose to the target pose may cause unreasonable interpenetration between the meshes of the human model. Only by combining colliders can we ensure that the meshes do not interpenetrate.

[0061] Combination Figures 1 to 3 The method for generating a three-dimensional human body model according to an embodiment of the present invention can be implemented by a human body image processing device. Figure 4 This is a schematic diagram illustrating the hardware structure 300 of a device for processing human body images according to an embodiment of the invention.

[0062] The present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the driving method and steps described above.

[0063] And an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements the driving method and steps described above.

[0064] like Figure 4 As shown, the human body fitting device 300 in this embodiment includes: a processor 301, a memory 302, a communication interface 303, and a bus 310, wherein the processor 301, the memory 302, and the communication interface 303 are connected through the bus 310 and complete communication between them.

[0065] Specifically, the processor 301 may include a central processing unit (CPU), an ASIC, or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0066] Memory 302 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 302 may include an HDD, floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where suitable, memory 302 may include removable or non-removable (or fixed) media. Where suitable, memory 302 may be internal or external to the human image processing device 300. In a particular embodiment, memory 302 is a non-volatile solid-state memory. In a particular embodiment, memory 302 includes read-only memory (ROM). Where suitable, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0067] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of the present invention.

[0068] Bus 310 includes hardware, software, or both, that couples components of device 300 for processing human images together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 310 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0069] In other words, Figure 4 The illustrated device 300 for processing human images can be implemented as including: a processor 301, a memory 302, a communication interface 303, and a bus 310. The processor 301, memory 302, and communication interface 303 are connected via the bus 310 and communicate with each other. The memory 302 stores program code; the processor 301 reads the executable program code stored in the memory 302 to run a program corresponding to the executable program code, thereby executing the three-dimensional human model fitting method in any embodiment of the present invention, and thus achieving the combination... Figures 1 to 3 The method and equipment for fitting three-dimensional human body models are described.

[0070] This invention also provides a computer storage medium storing computer program instructions; when executed by a processor, the computer program instructions implement the method for processing human images provided in this invention.

[0071] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0072] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0073] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0074] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A driving method of a mannequin, characterized by, The method comprises: 1) establishing a three-dimensional standard human body model in an initial posture, the three-dimensional standard human body model is self-established in combination with a mathematical model, and is composed of 20 body base parameters and 170 bone parameters, the 20 body bases constitute the whole human body model grid, each body base is individually controlled to change by a base parameter, and does not affect each other; 2) obtaining posture and body type parameters of a target three-dimensional human body model, the posture and body type parameters of the target three-dimensional human body model correspond to the bones and the 20 body base parameters of the three-dimensional standard human body model one by one; 3) inputting the obtained 20 bases and bone parameters into the standard three-dimensional human body model for fitting; 4) completing driving the bones to move from the initial posture to a target posture, comprising: a) obtaining position coordinates of the initial posture and the target posture, the initial posture parameters are determined by the initial parameters of the standard human body model, and the bone information of the target posture is obtained by regression prediction of a neural network model; b) generating an animation sequence from the initial posture to the target posture; c) after obtaining the initial state and the target posture state parameters of the bone information, forming a bone information time sequence from the initial posture to the target posture by linear interpolation or nearest neighbor interpolation interpolation frame method; d) in the driving process, the parent bone node is driven first, and then the child bone node is driven according to the number of bones driven per frame; e) the interpolation speed adopts a non-uniform interpolation rate, which is set to be slow at the positions away from the initial point and the target point, and fast in the middle movement process; f) in the process of generating the animation sequence, the driving speed is reduced when interpolation is performed to the end of fast movement, and the movement is static for several frames when moving to the final target posture, so as to obtain the whole animation sequence; 5) obtaining a three-dimensional target human body model grid with the same posture as a target human body two-dimensional image.

2. The method of claim 1, wherein, In the process of generating the animation sequence, the grid interpolation method is adopted for processing, after each frame of bone movement is driven, the current state human body model vertex, that is, the face information, is calculated by the weight parameters of the standard human body model, the current human body model grid state is updated and saved.

3. The method of claim 1, wherein, The standard human body model is a three-dimensional standard human body model constructed in combination with a mathematical model; the three-dimensional standard human body model has a mathematical weight relationship between the bone points and the model grid, and the determination of the bone points can be associated with the determination of the human body model of the target human body posture.

4. The method of claim 1, wherein, The step of obtaining the posture and body type parameters of the target three-dimensional human body model further comprises: 1) obtaining a two-dimensional image of the target human body; 2) processing the obtained two-dimensional human body contour image; 3) substituting the two-dimensional human body contour image into a first neural network subjected to deep learning to regress the joint nodes; 4) obtaining a joint node map of the target human body; obtaining a semantic segmentation map of each part of the human body; a body key point; a body bone point; 5) substituting the generated joint node map, semantic segmentation map, body bone point and key point information of the target human body into a second neural network subjected to deep learning to regress the human body posture and body type parameters; and 6) obtaining output three-dimensional human body parameters, including three-dimensional human body action posture parameters and three-dimensional human body body type parameters. ​ 5. The method of claim 4, wherein, Further comprising, the acquisition of the two-dimensional human body contour image utilizes a target detection algorithm, which is a target region fast generation network based on a convolutional neural network; before inputting the two-dimensional human body contour image into the first neural network model, further comprising a process of training the neural network, the training sample including a standard two-dimensional human body image with labeled original joint positions, which are labeled by human with high accuracy on the two-dimensional human body image.

6. A computer readable storage medium characterized by, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-5.

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-5.

Images