Human body model fitting method, device, and electronic equipment

Abstract

The present application provides a method, device and electronic device for fitting a human body model, the method comprising: obtaining an input first human body model; aligning the skeletal posture of a pre-stored second human body model with the skeletal posture of the first human body model; obtaining a sampling point set of the first human body model, the sampling point set including sampling points representing the posture of non-skeletal parts; determining sampling points corresponding to the designated parts of the second human body model from the sampling point set; aligning the posture of the designated parts of the second human body model according to the positions of the sampling points; aligning the body shape of the second human body model with the body shape of the first human body model. The above method can greatly improve the fitting accuracy of the local position of the human body model.

Description

Technical Field

[0001] The present application relates to the technical field of human body models, and in particular to a human body model fitting method, device and electronic equipment. Background Art

[0002] Driven by the wave of virtual reality technology, virtual digital humans are widely used in various industries, such as autonomous bank attendants, online education, virtual game anchors, etc. The implementation of virtual digital humans requires planning, human body modeling, binding animation, dubbing and other links. The human body model can be reconstructed through acquisition equipment, or three-dimensional human body fitting can be performed based on templates. However, due to the low accuracy of acquisition equipment and occlusion, the topology and model quality of the reconstructed human body model are poor; and the human body model obtained based on template fitting has problems such as low local detail fitting accuracy and semantic misalignment of human body parts. These are all technical problems that need to be solved at present. Summary of the invention

[0003] The present application provides a human body model fitting method, device and electronic equipment, which can effectively improve the fitting accuracy of local details of the model.

[0004] According to a first aspect of an embodiment of the present application, a human body model fitting method is provided, the method comprising:

[0005] Obtaining an input first human body model;

[0006] Aligning the pre-stored skeletal posture of the second human body model with the skeletal posture of the first human body model;

[0007] Acquire a sampling point set of the first human body model, wherein the sampling point set includes sampling points representing postures of non-skeletal parts;

[0008] Determining a sampling point corresponding to a designated part of the second human body model from the sampling point set;

[0009] aligning the posture of the designated part of the second human body model according to the position of the sampling point;

[0010] The body shape of the second human body model is aligned with the body shape of the first human body model.

[0011] According to a second aspect of an embodiment of the present application, a human body model fitting device is provided, the device comprising:

[0012] A model acquisition module, used for acquiring an input first human body model;

[0013] A skeleton posture alignment module, used to align the pre-stored skeleton posture of the second human body model with the skeleton posture of the first human body model;

[0014] A sampling point acquisition module, used to acquire a sampling point set of the first human body model, wherein the sampling point set includes sampling points representing postures of non-skeletal parts;

[0015] A determination module, configured to determine, from the sampling point set, a sampling point corresponding to a designated part of the second human body model;

[0016] A designated part posture alignment module, used for aligning the posture of the designated part of the second human body model according to the position of the sampling point;

[0017] A body shape alignment module is used to align the body shape of the second human body model with the body shape of the first human body model.

[0018] According to a third aspect of an embodiment of the present application, an electronic device is provided, the electronic device comprising:

[0019] processor;

[0020] a memory for storing processor-executable instructions;

[0021] Wherein, the processor is configured to:

[0022] Obtaining an input first human body model;

[0023] Aligning the pre-stored skeletal posture of the second human body model with the skeletal posture of the first human body model;

[0024] Acquire a sampling point set of the first human body model, wherein the sampling point set includes sampling points representing postures of non-skeletal parts;

[0025] Determining a sampling point corresponding to a designated part of the second human body model from the sampling point set;

[0026] aligning the posture of the designated part of the second human body model according to the position of the sampling point;

[0027] The body shape of the second human body model is aligned with the body shape of the first human body model.

[0028] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0029] The human body model fitting method provided in the present application first aligns the skeletal posture of an input first human body model with a pre-stored second human body model, then uses a sampling point set to obtain detailed information of the first human body model, and based on this, aligns the postures of designated parts in the first and second human body models, and at the same time aligns the body shapes of the two models, thereby greatly improving the fitting accuracy of the local position of the human body model.

[0030] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0032] Figure 1 It is a flow chart of a human body model fitting method shown in the present application according to an exemplary embodiment.

[0033] Figure 2 It is a sampling point of a fingertip of a hand in a first human body model according to another exemplary embodiment of the present application.

[0034] Figure 3 It is a schematic diagram of the alignment effect of hand fingertips and hand postures shown in the present application according to another exemplary embodiment.

[0035] Figure 4 It is a schematic diagram of the hand posture and body shape fitting effect shown in accordance with another exemplary embodiment of the present application.

[0036] Figure 5 It is a schematic diagram of the posture and body shape fitting effect of a human model according to another exemplary embodiment of the present application.

[0037] Figure 6 is a flow chart of a human body fitting method according to another exemplary embodiment of the present application.

[0038] Figure 7 It is a block diagram of a human body fitting device shown in the present application according to an exemplary embodiment.

[0039] Figure 8 It is a hardware structure diagram of an electronic device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0040] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0041] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0042] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0043] Next, the embodiments of the present application are described in detail.

[0044] The human body model can be reconstructed through various acquisition devices, such as using a scanner to scan and measure the real human body from multiple angles, and then reconstructing a three-dimensional human body model that is consistent with the real human body shape and posture after obtaining the parameters of various parts of the human body. However, due to the low accuracy of the acquisition equipment, or occlusion by clothes, the topology and quality of the human body model reconstructed by scanning with the acquisition equipment are poor. In addition, the human body model can also be fitted with a three-dimensional human body based on a template, but due to problems such as low accuracy in fitting local details and misalignment of human body semantics, the fitting results are poor. There are various problems with the above two methods for obtaining human bodies, which result in the obtained human body models requiring a lot of post-processing manpower before they can be put into use. In order to at least partially solve the above technical problems, the present application provides a human body model fitting method, such as Figure 1 The following steps are included:

[0045] Step 110: Obtain an input first human body model;

[0046] Step 120: aligning the pre-stored skeletal posture of the second human body model with the skeletal posture of the first human body model;

[0047] Step 130: Acquire a sampling point set of the first human body model, wherein the sampling point set includes sampling points representing postures of non-skeletal parts;

[0048] Step 140: determining a sampling point corresponding to a designated part of the second human body model from the sampling point set;

[0049] Step 150: aligning the posture of the designated part of the second human body model according to the position of the sampling point;

[0050] Step 160: Align the body shape of the second human body model with the body shape of the first human body model.

[0051] The human body model fitting method provided in the present application first aligns the skeletal posture of an input first human body model with a pre-stored second human body model, then uses a sampling point set to obtain detailed information of the first human body model, and based on this, aligns the postures of designated parts in the first and second human body models, and at the same time aligns the body shapes of the two models, thereby greatly improving the fitting accuracy of the local position of the human body model.

[0052] In some embodiments, the first human body model input in the above step 110 can be a human body model reconstructed by scanning a real human body at multiple angles through an acquisition device such as a scanner. In addition to reconstruction through hardware equipment, the first human body model can also be a human body model generated based on computer vision technology or artificial intelligence AI technology. This application does not limit the generation method of the first human body model, and human body models obtained by various other methods can be used as the first human body model of this application.

[0053] In addition, the second human body model pre-stored in step 120 can be obtained from a pre-established human body model library according to the input human body characteristic parameters. Among them, the establishment of the human body model library can use the trained parameterized human body model. By adjusting the body shape parameters of the parameterized human body model, a human body model library of different body shapes and genders in the same posture is established, and abnormal data that does not conform to the statistical distribution of the human body is removed. In some embodiments, the human body models in the human body model library carry various parameters and information, such as body shape parameters, position information of various parts of the human body, bone point information, etc. The human body models in the human body model library are actually trained parameterized human body models. Compared with the human body models reconstructed by scanning, their topology and model quality are higher, and they are suitable for subsequent use and development of the model. Subsequently, according to the input human body characteristic parameters, including but not limited to height, weight and gender, the second human body model is obtained from the human body model library. In some embodiments, the human body characteristic parameters can also include the size or proportion of some specific parts of the human body, such as measurements, head-to-body ratio, etc. By inputting the human body characteristic parameters, a second human body model with initial parameters can be obtained from the human body model library.

[0054] After obtaining the first and second human models, step 120 is executed to align the skeletal posture of the second human model with the skeletal posture of the first human model. In some embodiments, the skeletal points of the second human model can be adjusted to align the skeletal posture of the second human model with the skeletal posture of the first human model. Skeletal points are data points that characterize the joints of the human body. Usually, 10 to 12 skeletal points can be used to effectively describe the action posture of the human body. As described above, the human models in the human model library carry skeletal point information, and the skeletal posture of the second human model can be obtained based on these skeletal point information. The input first human model may carry skeletal point information or skeletal position, or it may not carry it. Therefore, in some embodiments, before aligning the skeletal posture, it is also necessary to obtain the skeletal posture of the first human model, including detecting the key points of the first human model, generating corresponding skeletal points according to the key points, and obtaining the skeletal posture of the first human model. Usually, the input first human model is first rendered 360° to obtain a rendered image of the first human model at a certain perspective, and then the rendered image is key-point detected based on a two-dimensional or three-dimensional human key point detection algorithm to obtain the corresponding three-dimensional key point information. Finally, according to the semantic information of the skeleton, the correspondence between the detected three-dimensional key points and the skeleton points is established to generate the corresponding skeleton points. The skeleton points can be key points after position optimization, or can be obtained by interpolating the key point positions. The above provides a skeleton point generation method for one of the human body models. Those skilled in the art can select other skeleton point generation methods according to actual needs, and this application is not limited here.

[0055] The above-mentioned skeletal points can represent the postures of the head, trunk, limbs and other parts of the human body. Therefore, after the skeletal postures are aligned, the postures of the head, trunk, limbs and other parts of the first human body model and the second human body model have been roughly aligned. However, for the parts that cannot be represented by the skeletal points, such as hands, feet, etc., their postures are still not aligned. Therefore, the present application proposes to use a sampling point set that can represent more human body model information to align the parts that cannot be represented by the skeletal points. Among them, the sampling point set includes sampling points that represent the postures of non-skeletal parts, and of course, it can also include sampling points that represent the postures of skeletal parts. In some embodiments, the sampling point set can be obtained by resampling the surface of the first human body model based on a heat map algorithm. Of course, the sampling point set of the surface of the first human body model can also be obtained based on RGB images or using the key point detection method of a depth camera. Then, the sampling point corresponding to the specified part of the second human body model is determined from the sampling point set. The specified part can be a non-skeletal part, that is, a part that cannot be represented by the skeletal points, such as the fingertips of the hands or feet. After obtaining the sampling point set on the surface of the first human body model, it is necessary to determine the position of the designated part in the first human body model, or in other words, it is necessary to determine which sampling points in the sampling point set can represent the designated part. The labeling of the sampling points of the designated part can be done manually, or the position of the designated part in the second human body model can be used to select sampling points that match the position from the sampling point set as sampling points corresponding to the designated part. As described above, the human body models in the human body model library can carry position information of various parts of the human body. After the first human body model and the second human body model are overlapped and aligned, the sampling point with the shortest distance from the position of the designated part of the second human body model can be used as the sampling point corresponding to the designated part based on the position information of the designated part in the second human body model. The determined sampling points can represent the position and posture of the designated part in the first human body model. Figure 2 As shown, when the designated part is the fingertip of the hand, according to the position of the fingertip of the hand on the second human body model (not shown in the figure), the sampling point corresponding to the fingertip of the hand can be determined from the sampling point set (such as Figure 2 The position of each sampling point can accurately identify each fingertip, that is, the position of each finger, and the distance between the sampling points can characterize the degree of opening and closing between the fingers, that is, the hand posture.

[0056] After determining the sampling point corresponding to the designated part from the sampling point set, that is, the position and posture of the designated part in the first human body model, the posture of the designated part of the second human body model is aligned according to the position of the sampling point. In some embodiments, the posture alignment of the designated part can be performed based on a position transformation matrix, which is used to characterize the mapping relationship between the position of the sampling point and the position of the designated part in the second human body model. Using the above-detected position of the designated part in the first human body model and the position of the designated part in the second human body model, the position transformation matrix can be determined. And based on the position transformation matrix, the posture alignment of the designated part and the part of the human body where the designated part is located is performed. For example, in the above example, when the designated part is a fingertip of the hand, such as Figure 3 As shown, the effect comparison before and after the hand posture adjustment is performed. The position transformation matrix is ​​determined by using the position of the hand fingertips in the first person and second body models, and the posture of the hand fingertips and the hand are aligned based on the position transformation matrix.

[0057] After the above alignment steps, the posture of the second human body model, whether it is the skeletal part or the non-skeletal part, has been aligned with the posture of the first human body model. In addition to posture alignment, the body shape of the human body model also needs to be aligned, that is, the body shape of the second human body model is aligned with the body shape of the first human body model. In some embodiments, the body shape alignment can be performed based on the body shape parameters of the first human body model, and the body shape parameters can be obtained by parsing the first human body model. When executing step 110 to obtain the input first human body model, it is necessary to parse the first human body model, and at this time, several body shape parameters of the first human body model can be obtained. In addition to being related to height, weight, and gender, a person's body shape is also related to muscle distribution. Even if two people of the same height, weight, and gender, their body shapes will not be exactly the same. Therefore, it is also necessary to adjust the body shape of the second human body model according to the body shape parameters carrying body shape information such as muscle distribution of the first human body model, so that the body shapes of the first and second human body models are aligned. In some embodiments, the body shape alignment based on body shape parameters can be performed simultaneously with the skeletal posture alignment. An optimization equation with the goal of aligning body shape and posture can be established, and body shape parameters and bone positions can be optimized through step-by-step iteration, and the final aligned body shape parameters and bone positions can be obtained. According to the human body model optimized at each step in the iteration, the bone position in the second human body model is regressed and then the body shape parameters are optimized again until the optimization is completed, and a second human body model whose body shape and posture are aligned with the first human body model is obtained. The body shape alignment based on body shape parameters and the bone posture alignment are performed simultaneously, and the second human body model whose body shape and posture are basically aligned with the first human body model can be obtained before the posture alignment of the designated part is performed.

[0058] The sampling point set on the surface of the first model can not only characterize the posture of the non-skeletal parts, but also characterize the body shape of the non-skeletal parts, and of course, can also characterize the body shape of the skeletal parts. Therefore, in addition to being based on the above-mentioned body shape parameters, body shape alignment can also be performed based on the sampling point set. Since the sampling points can characterize the body shapes of the skeletal parts and non-skeletal parts in the human body model, the body shape alignment based on the sampling point set can align the body shapes of the second human body model including the skeletal parts and non-skeletal parts. Since the body shape parameters can only characterize the body shape characteristics of some human body parts, for some parts such as hands and feet that cannot be characterized by body shape parameters, it is impossible to optimize the alignment of the shapes of these parts by adjusting the body shape parameters. Compared with body shape parameters, the sampling points on the surface of the human body model can characterize the body shape of the entire human body model, so body shape alignment can be performed on parts that cannot be characterized by body shape parameters. The body shape alignment of the second human body model based on the sampling point set can be constrained deformation of the second human body model.

[0059] In some embodiments, before performing posture alignment on the designated part, the second human body model body shape may be aligned based on body shape parameters, and after obtaining the sampling point set of the first human body model, body shape alignment with higher accuracy may be performed based on the sampling point set. Figure 4-5 As shown, the left picture is the input first human body model, and the right picture is the model fitted by the second human body model through the above method. By comparing the left and right pictures, it can be seen that after the second human body model is fitted, the posture and shape of the torso, limbs, hands and other local parts are consistent with the first human body model, achieving a high degree of fitting.

[0060] The human body model fitting method provided in the present application first aligns the skeletal posture of an input first human body model with a pre-stored second human body model, then uses a sampling point set to obtain detailed information of the first human body model, and based on this, aligns the postures of designated parts in the first and second human body models, and at the same time aligns the body shapes of the two models, thereby greatly improving the fitting accuracy of the local position of the human body model.

[0061] In addition, the present application also provides a human body model fitting method, including: Figure 6 Steps shown:

[0062] Step 611: obtaining an input first human body model and obtaining body shape parameters of the first human body model;

[0063] Step 612: Detect key points of the first human body model, generate corresponding skeleton points according to the key points, and obtain the skeleton posture of the first human body model;

[0064] Step 620: Acquire a second human body model from a pre-established human body model library; the second human body model carries body shape parameters and skeletal posture information;

[0065] There is no time relationship between the execution of steps 611-612 and step 620. After executing steps 611-612 and step 620, execute:

[0066] Step 630: aligning the second human body model with the first human body model in terms of skeletal posture; and aligning the body shape based on body shape parameters;

[0067] Step 640: resampling the surface of the first human body model based on a heat map algorithm to obtain a sampling point set;

[0068] Step 650: determining a sampling point corresponding to a designated part of the second human body model from the sampling point set;

[0069] Step 660: aligning the posture of the designated part of the second human body model according to the position of the sampling point;

[0070] Step 670: The body shape of the second human body model is aligned with the first human body model based on the sampling point set.

[0071] The specific implementation methods of the above steps refer to the above embodiments, and this application will not go into details here.

[0072] Based on the human body model fitting method described in any of the above embodiments, the present application also provides Figure 7 The human body fitting device 700 shown includes:

[0073] A model acquisition module 710 is used to acquire an input first human body model;

[0074] A skeleton posture alignment module 720, used to align the pre-stored skeleton posture of the second human body model with the skeleton posture of the first human body model;

[0075] A sampling point acquisition module 730 is used to acquire a sampling point set of the first human body model, wherein the sampling point set includes sampling points representing postures of non-skeletal parts;

[0076] A determination module 740 is used to determine a sampling point corresponding to a designated part of the second human body model from the sampling point set;

[0077] A designated part posture alignment module 750, used to align the posture of the designated part of the second human body model according to the position of the sampling point;

[0078] The body shape alignment module 760 is used to align the body shape of the second human body model with the body shape of the first human body model.

[0079] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0080] Based on the human body model fitting method described in any of the above embodiments, the present application also provides Figure 8 A schematic diagram of the structure of an electronic device is shown in FIG. Figure 8 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the human body model fitting method described in any of the above embodiments.

[0081] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0082] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

Claims

1. A human body model fitting method, characterized in that: The method comprises: Obtaining an input first human body model; Aligning the pre-stored skeletal posture of the second human body model with the skeletal posture of the first human body model; Acquire a sampling point set of the first human body model, wherein the sampling point set includes sampling points representing postures of non-skeletal parts; Determining a sampling point corresponding to a designated part of the second human body model from the sampling point set; Aligning the posture of the designated part of the second human body model based on a position transformation matrix, wherein the position transformation matrix represents a mapping relationship between the position of the sampling point and the position of the designated part on the second human body model; The body shape of the second human body model is aligned with the body shape of the first human body model.

2. The method according to claim 1, characterized in that The pre-stored second human body model is obtained from a pre-established human body model library according to the input human body characteristic parameters.

3. The method according to claim 2, characterized in that The human body characteristic parameters include at least one of the following: height, weight and gender.

4. The method according to claim 1, characterized in that: Before aligning the skeleton posture, the method further includes: detecting key points of the first human body model, generating corresponding skeleton points according to the key points, and acquiring the skeleton posture of the first human body model.

5. The method according to claim 1, characterized in that: The sampling point set is obtained by resampling the surface of the first human body model based on a heat map algorithm.

6. The method according to claim 1, characterized in that The sampling points corresponding to the designated parts of the second human body model are determined by the following steps: According to the position of the designated part on the second human body model, sampling points matching the position are selected from the sampling point set as sampling points corresponding to the designated part.

7. The method according to claim 1, characterized in that The designated part includes fingertips of hands or feet.

8. The method according to claim 1, characterized in that The body shape alignment is performed based on body shape parameters of the first human body model, and the body shape parameters are acquired by analyzing the first human body model.

9. The method according to claim 1, characterized in that: The sampling point set also includes sampling points that characterize the body shape of the non-skeletal part; the alignment of the body shape of the second human body model is performed based on the sampling point set.

10. A human body model fitting device, characterized in that: The device comprises: A model acquisition module, used to acquire an input first human body model; A skeleton posture alignment module, used for aligning the pre-stored skeleton posture of the second human body model with the skeleton posture of the first human body model; A sampling point acquisition module, used to acquire a sampling point set of the first human body model, wherein the sampling point set includes sampling points representing postures of non-skeletal parts; A determination module, configured to determine, from the sampling point set, a sampling point corresponding to a designated part of the second human body model; A designated part posture alignment module, used for aligning the posture of the designated part of the second human body model based on a position transformation matrix, wherein the position transformation matrix represents a mapping relationship between the position of the sampling point and the position of the designated part on the second human body model; A body shape alignment module is used to align the body shape of the second human body model with the body shape of the first human body model.

11. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Obtaining an input first human body model; Aligning the pre-stored skeletal posture of the second human body model with the skeletal posture of the first human body model; Acquire a sampling point set of the first human body model, wherein the sampling point set includes sampling points representing postures of non-skeletal parts; Determining a sampling point corresponding to a designated part of the second human body model from the sampling point set; Aligning the posture of the designated part of the second human body model based on a position transformation matrix, wherein the position transformation matrix represents a mapping relationship between the position of the sampling point and the position of the designated part on the second human body model; The body shape of the second human body model is aligned with the body shape of the first human body model.

Images