Image acquisition method and device and related equipment

CN119027212APending Publication Date: 2024-11-26MIGU CO LTD +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411074134.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the existing virtual fitting technology, due to the large differences in body shape and movement of the deformation algorithm, the generated virtual fitting image has poor effect.

Method used

By estimating the target user image and the target clothing image, the corresponding relationship with the target three-dimensional human model is obtained, parametric three-dimensional human modeling is carried out, clothing texture images are migrated, and image repair and noise reduction are used to generate More realistic virtual fitting images.

Benefits of technology

The effect of virtual fitting is improved, ensuring the accurate migration and realism of clothing texture, and solving the problem of poor virtual fitting pictures caused by body shape and movement differences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027212A_ABST
    Figure CN119027212A_ABST
Patent Text Reader

Abstract

The invention provides an image acquisition method and device and related equipment, and the method comprises the steps: carrying out the posture estimation of a target user image and a target clothes image, and obtaining a first corresponding relation and a second corresponding relation; performing parameterized three-dimensional human body modeling on the first corresponding relation and the second corresponding relation by using a target three-dimensional human body model to obtain a first texture image and a second texture image; and migrating the second texture image into the first texture image, and generating a target image of the target user wearing the target clothing. According to the method, the first corresponding relation and the second corresponding relation relative to the target three-dimensional human body model are obtained after the target user image and the target clothing image are subjected to posture estimation, the first texture image and the second texture image are obtained after modeling, and the second texture image is migrated into the first texture image; and obtaining the third image of the image of the target user wearing the target clothing, thereby improving the effect of virtual fitting.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular, to a method, apparatus and related equipment for image acquisition. Background Art

[0002] With the development of artificial intelligence generation capabilities, people hope to be able to simulate or generate some virtual scenes through artificial intelligence. Virtual fitting is one of the scenes that has attracted much attention. People hope to see the clothes on the model on themselves immediately, or they are not satisfied with the clothes in their photos and hope to change them. In the existing technology, most virtual fitting is based on deformation algorithms. Such deformation algorithms may have large differences in body shape and movement for different fitting pictures, resulting in poor results of the generated virtual fitting pictures. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus and related equipment for acquiring an image to solve the problem of poor effect of virtual fitting images generated in the prior art.

[0004] To solve the above problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for acquiring an image, the method comprising:

[0006] Performing posture estimation on the target user image and the target clothing image respectively to obtain a first corresponding relationship and a second corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the target user and the target three-dimensional human body model, and the second corresponding relationship is a corresponding relationship between the target model and the target three-dimensional human body model;

[0007] Using the target three-dimensional human body model, respectively, parameterized three-dimensional human body modeling is performed on the first corresponding relationship and the second corresponding relationship to obtain a first texture image and a second texture image;

[0008] The second texture image is transferred to the first texture image to generate a target image of the target user wearing the target clothing.

[0009] Optionally, performing posture estimation on the target user image and the target clothing image respectively to obtain a first corresponding relationship and a second corresponding relationship includes:

[0010] Inputting the target user image and the target clothing image into a human body two-dimensional posture estimation model respectively, and outputting a plurality of first feature maps and a plurality of second feature maps, wherein the plurality of first feature maps respectively include a plurality of body parts of the target user, and the plurality of second feature maps respectively include a plurality of body parts of the model corresponding to the target clothing image;

[0011] Based on the target convolutional neural network, three-dimensional posture estimation is performed on the multiple first feature maps and the multiple second feature maps respectively to obtain the first corresponding relationship and the second corresponding relationship. The target convolutional neural network is generated based on the target three-dimensional human body model.

[0012] Optionally, the step of migrating the second texture image to the first texture image to generate a target image of the target user wearing the target clothing includes:

[0013] Filling the pixel values ​​on the second texture image into a standard texture expansion image to obtain a third texture image, wherein the standard texture expansion image is obtained by expanding a standard human body three-dimensional model;

[0014] Determine an offset value according to the target clothing image, wherein the offset value is used to indicate parameter information during image migration;

[0015] Migrating the third texture image to the first texture image according to the offset value to generate a migrated image;

[0016] The migration image is input into a conditional diffusion model for image repair and image denoising to obtain the target image.

[0017] Optionally, determining the offset value according to the target clothing image includes:

[0018] Decomposing the target clothing image to obtain a diffuse color texture map, a normal vector texture map, a highlight texture map, and a roughness texture map;

[0019] Inputting the diffuse color texture map, the normal vector texture map, the highlight texture map, and the roughness texture map into an encoder to obtain a plurality of low-dimensional codes;

[0020] averaging the vectors corresponding to the multiple low-dimensional codes to obtain the material information of the target clothing;

[0021] The offset value is generated according to the material information.

[0022] Optionally, generating the offset value according to the material information includes:

[0023] Segmenting the target clothing image by a semantic segmentation model to obtain a plurality of clothing regions, wherein different clothing regions correspond to different positions of the target clothing;

[0024] Projecting the target three-dimensional human body model on a target plane to obtain a target vector, wherein the target plane is a camera plane of the target three-dimensional human body model;

[0025] Determine at least one target clothing area among the multiple clothing areas according to the target vector, the target clothing area being a clothing area among the multiple clothing areas that intersects with the target vector;

[0026] The at least one target clothing area is input into the displacement model to obtain the offset value, and the displacement model is used to calculate the displacement offset of the at least one target clothing area on the target user image.

[0027] Optionally, before performing parameterized 3D human body modeling on the first correspondence and the second correspondence respectively using the target 3D human body model to obtain a first texture image and a second texture image, the method further includes:

[0028] Acquire a training data set, wherein the training data set includes a plurality of human body images;

[0029] Inputting the training data set into a regression model, and outputting camera parameters, motion parameters, and body shape parameters, wherein the camera parameters are parameters of a camera used to shoot the plurality of human body images, and the motion parameters and the body shape parameters are parameters constituting the target three-dimensional human body model;

[0030] The three-dimensional human body model is updated according to the camera parameters, the action parameters and the body shape parameters to obtain the target three-dimensional human body model.

[0031] In a second aspect, an embodiment of the present application further provides an image acquisition device, the device comprising:

[0032] an estimation module, configured to perform posture estimation on the target user image and the target clothing image respectively, to obtain a first corresponding relationship and a second corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the target user and the target three-dimensional human body model, and the second corresponding relationship is a corresponding relationship between the target model and the target three-dimensional human body model;

[0033] A modeling module, used for performing parameterized three-dimensional human body modeling on the first corresponding relationship and the second corresponding relationship respectively by using the target three-dimensional human body model to obtain a first texture image and a second texture image;

[0034] A generating module is used to transfer the second texture image to the first texture image to generate a target image of the target user wearing the target clothing.

[0035] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps of the method described in the first aspect above.

[0036] In a fourth aspect, an embodiment of the present application further provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the method described in the first aspect above.

[0037] In a fifth aspect, an embodiment of the present application further provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps in the method described in the first aspect.

[0038] The present application provides a method, device and related equipment for image acquisition, the method comprising: performing posture estimation on a target user image and a target clothing image respectively to obtain a first corresponding relationship and a second corresponding relationship, the first corresponding relationship being the corresponding relationship between the target user and a target three-dimensional human body model, and the second corresponding relationship being the corresponding relationship between the target model and the target three-dimensional human body model; using the target three-dimensional human body model to perform parameterized three-dimensional human body modeling on the first corresponding relationship and the second corresponding relationship respectively to obtain a first texture image and a second texture image; migrating the second texture image to the first texture image to generate a target image of the target user wearing the target clothing. The present application performs posture estimation on the target user image and the target clothing image to obtain a first corresponding relationship and a second corresponding relationship relative to the target three-dimensional human body model, thereby obtaining a first texture image and a second texture image after modeling, migrating the second texture image to the first texture image to obtain a third image of the target user wearing the target clothing, thereby improving the effect of virtual fitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 It is a flowchart of the method for image acquisition provided in an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of image processing provided by an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of the structure of an image acquisition device provided in an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The terms "first", "second" etc. in the embodiments of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. In addition, the terms "include" and "have" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, "and / or" is used in the present application to represent at least one of connected objects, such as A and / or B and / or C, representing the inclusion of separate A, separate B, separate C, and A and B all exist, B and C all exist, A and C all exist, and 7 situations in which A, B and C all exist.

[0046] See also Figure 1 , Figure 1 is a flow chart of the method for image acquisition provided in the embodiment of the present application, such as Figure 1 As shown, the method of image acquisition includes:

[0047] Step 101: perform posture estimation on a target user image and a target clothing image respectively to obtain a first correspondence relationship and a second correspondence relationship, wherein the first correspondence relationship is a correspondence relationship between the target user and a target three-dimensional human body model, and the second correspondence relationship is a correspondence relationship between the target model and the target three-dimensional human body model.

[0048] In this embodiment, the target user image is a human body image of the target user to be tried on, and the target clothing image is an image of the target model wearing the target clothing. The target user image and the target clothing image are respectively subjected to posture estimation through human posture estimation to obtain a first corresponding relationship and a second corresponding relationship. Among them, the posture estimation is used to determine the corresponding relationship between the person in the image and the target three-dimensional human body model, that is, to obtain a parametric representation of the person on the target three-dimensional human body model. Specifically, the target three-dimensional human body model is a parametric three-dimensional human body model SMPL, wherein SMPL is a widely used three-dimensional human body representation. In SMPL, a standard average three-dimensional human body model is first defined, and the three-dimensional model consists of 6890 vertices and 13776 facets. On the basis of the standard human body, two parameter vectors are defined, which are a 10-dimensional parameter vector for representing the human body shape, called body shape parameters, and a key point rotation vector for representing human body movements, called action parameters. Based on the study of a large number of three-dimensional human bodies, the SMPL parametric model can describe the displacement of the vertices on the human body surface due to the influence of body shape and body movements on the basis of a standard human body. Therefore, under this model, only two parameter vectors, the position of the key points of the human body and the body shape, need to be given to obtain the corresponding three-dimensional human body point cloud.

[0049] It should be noted that the first correspondence and the second correspondence refer to a unified representation in a parameterized three-dimensional human body model, so that the clothing texture in the model image that overlaps with the human body can be reasonably migrated to the target human body image. That is, for a two-dimensional image, the human body (target human body or model) in the image is fitted to the parameterized three-dimensional human body model, and the camera parameters when the image is taken are estimated, so that the pixel points on the two-dimensional image can be matched one by one with the vertices on the surface of the three-dimensional model through the projection matrix. Through this correspondence between two-dimensional and three-dimensional, the correspondence between the position of the vertex of the clothing part and the human body surface to which it is attached in the clothing image can be obtained, and because a unified SMPL parameterized three-dimensional human body model is used, the vertex numbers on the surface of the three-dimensional human body model of the target human body and the three-dimensional human body model of the model are consistent, so that the clothing texture can be migrated to the target human body.

[0050] Step 102: Use the target three-dimensional human body model to perform parameterized three-dimensional human body modeling on the first corresponding relationship and the second corresponding relationship respectively to obtain a first texture image and a second texture image.

[0051] In this embodiment, the target three-dimensional human body model is used to perform parameterized three-dimensional human body modeling on the first correspondence relationship and the second correspondence relationship obtained above, respectively, so as to obtain a first texture image and a second texture image, respectively. Specifically, the parameterized three-dimensional human body modeling is used to model the first correspondence relationship and the second correspondence relationship obtained by parameterization to obtain a texture image, and illustratively, for example, SMPL point clouds corresponding to the target user image and the target clothing image are generated. It should be noted that the first texture image and the second texture image represent the SMPL point clouds corresponding to the target user image and the target clothing image, respectively.

[0052] Step 103: Migrate the second texture image to the first texture image to generate a target image of the target user wearing the target clothing.

[0053] In this embodiment, since most of the clothes are deformed to obtain the images after fitting, such as through spline interpolation, optical flow method, etc., the problem of whether the unique texture of the clothes can be maintained is not considered. For example, the texture of some fabrics is unique. In the case of a large difference in body shape between the model and the human body, this type of deformation algorithm will cause the texture to be flattened or deformed seriously, affecting the realism of the fitting image. In addition, the corresponding texture map only maps the texture of the overlapped part of the image with the estimated human body, and is close to the surface of the human body. The clothes do not have a sense of reality and three-dimensionality, which is not in line with common sense. For example, wrinkles are generated due to different human body movements, so it is necessary to correct the obtained texture map to generate an offset value, and the target user image and the target clothing image are migrated through the offset value, so as to accurately generate a third image.

[0054] The present application obtains a first correspondence and a second correspondence relative to a target three-dimensional human body model after performing posture estimation on a target user image and a target clothing image, thereby obtaining a first texture image and a second texture image after modeling, and migrating the second texture image to the first texture image to obtain a third image of the target user wearing the target clothing, thereby improving the effect of virtual fitting.

[0055] In some feasible implementations, optionally, performing posture estimation on the target user image and the target clothing image respectively to obtain the first corresponding relationship and the second corresponding relationship includes:

[0056] Inputting the target user image and the target clothing image into a human body two-dimensional posture estimation model respectively, and outputting a plurality of first feature maps and a plurality of second feature maps, wherein the plurality of first feature maps respectively include a plurality of body parts of the target user, and the plurality of second feature maps respectively include a plurality of body parts of the model corresponding to the target clothing image;

[0057] Based on the target convolutional neural network, three-dimensional posture estimation is performed on the multiple first feature maps and the multiple second feature maps respectively to obtain the first corresponding relationship and the second corresponding relationship. The target convolutional neural network is generated based on the target three-dimensional human body model.

[0058] In this embodiment, a two-dimensional posture estimation model is used to extract features from an image containing a person to obtain a feature image. Exemplarily, the two-dimensional posture estimation model can be an Openpose framework. Specifically, the positions of key points of the image are first estimated through the Openpose framework, wherein Openpose is a real-time human two-dimensional posture estimation model based on a convolutional deep network, which extracts features from an input image containing a human body through a convolutional network to obtain a set of feature maps, namely, multiple first feature maps and multiple second feature maps. Among them, the multiple first feature maps correspond to multiple body parts of the target user, and the multiple second feature maps correspond to multiple body parts of the model, and it should be noted that the number of the first feature maps and the second feature maps is the same, and the corresponding first feature maps and second feature maps correspond to the same body parts.

[0059] Then, two branches are used to extract the part position information map and the part affinity map, detect the positions of different parts of the human body and connect them to get a complete human skeleton and the positions of pre-defined human key points. In this step, the image containing the human body is input into Openpose, and the model will output the estimated positions of 25 key points of the human body in the model image. The positions of the joint points will help calculate the ratio of the length of the clothing to the length of the human skeleton, which is used to adapt to the new target human body.

[0060] The target convolutional neural network is the Densepose framework. Specifically, Densepose uses a convolutional deep network with pyramid network features to match the pixel points of the human body in the input two-dimensional image with the face element vertices on the three-dimensional human body model one by one. The process is to divide the face elements on the three-dimensional human body model into different regions according to different human body parts, expand the different regions to obtain UV texture maps, transform the predicted spatial coordinates of the human body surface, obtain the corresponding UV coordinates and record them on the corresponding pixel points of the two-dimensional image. In addition to the UV coordinates, the pixel points of the two-dimensional image will also record the body part label to which the area belongs. As shown in the figure below, the human body surface will be divided into 24 different regions.

[0061] It should be noted that the Densepose framework used in this step is actually obtained with the help of the parameterized 3D human body model SMPL, which is a widely used 3D human body representation. As shown in the following formula, is the standard average 3D human body model, β is the body shape parameter, θ is the motion parameter, Bs and B p It is the body shape and motion displacement matrix obtained by statistical learning of a large amount of human body data. On the basis of the average human body model, the displacement caused by body shape and motion is considered to obtain the final position T of all vertices on the three-dimensional human body.

[0062]

[0063] Densepose predicts the correspondence of the texture map based on the SMPL model on the two-dimensional image. From the texture coordinates of each pixel point in the two-dimensional human body image in its corresponding area, we can further get its corresponding vertex on the three-dimensional human body model. This application proposes to use the relationship between the two to realize virtual fitting of any body shape.

[0064] Optionally, the step of migrating the second texture image to the first texture image to generate a target image of the target user wearing the target clothing includes:

[0065] Filling the pixel values ​​on the second texture image into a standard texture expansion image to obtain a third texture image, wherein the standard texture expansion image is obtained by expanding a standard human body three-dimensional model;

[0066] Determine an offset value according to the target clothing image, wherein the offset value is used to indicate parameter information during image migration;

[0067] Migrating the third texture image to the first texture image according to the offset value to generate a migrated image;

[0068] The migration image is input into a conditional diffusion model for image repair and image denoising to obtain the target image.

[0069] In this embodiment, if Figure 2As shown, by estimating the input image, the SMPL point clouds corresponding to the target user image and the target clothing image are obtained respectively. Specifically, the texture coordinates in Densepose are obtained with the help of the SMPL model, so the corresponding XYZ coordinates can be calculated from its texture UV coordinates. However, since the target human body image and the target clothing model image do not belong to the same three-dimensional human body, a unified representation is required to correspond the two. Since the vertex serial number of SMPL is unchanged, such as the serial numbers of the vertices located at the eyes on different three-dimensional human bodies are consistent, they can be used as connections. It should be noted that the standard human body three-dimensional model is a predefined human body three-dimensional model, and the standard human body three-dimensional model is used to generate a standard texture expansion image, and the pixel values ​​of the second texture image can be filled by the standard texture expansion image. In this embodiment, the specific definition method of the standard human body three-dimensional model is not specifically limited. For example, a standard SMPL three-dimensional human body expansion texture map is defined, and the uv coordinates of each vertex on the texture map are recorded in the form of an index, that is, {vid: (u, v), vid∈[0,6889]}, where vid represents the vertex serial number and (u, v) represents the coordinates on the unified texture map. After using Densepose to estimate the model image, the iuv value of a pixel point p is (p i ,p u ,p v ), and then use the regression network to estimate the parameters of the SMPL human body model corresponding to the graph, and obtain the position of each vertex on the model {(v x ,v y ,v z ) vid ,vid∈[0,6889]}, and calculate its xyz coordinates through the correspondence between densepose and SMPL as (p x ,p y ,p z ), according to the above index and the interpolation calculation of the corresponding coordinates, the pixel values ​​on the model image can be filled into the standard texture expansion map to obtain the standard texture expansion map corresponding to the image. The same calculation is performed on the human body image. According to the index and the interpolation of the coordinate values, the pixel value corresponding to each vertex on the texture map is obtained to obtain the preliminary clothing texture migration result from the model image to the human body image.

[0070] In this embodiment, a conditional diffusion model is used as a completion synthesis network because the repair ability of the diffusion model is very powerful. Because the goal is to generate a real fitting image, only real data is used for training here. First, the real data is randomly destroyed, such as adding noise, random masking, etc., to obtain an incomplete clothing wearing picture, which is used to simulate the problem to be solved. The goal of the network is to refer to the key point positions of the human body and the model picture, and to complete and reduce the noise of the damaged clothing wearing picture, including the completion of the human body, so that the output image is consistent with that before it is destroyed. Thus, the image is modified and denoised by the conditional diffusion model, so that the migration image is repaired to obtain the target image. Specifically, in the actual application process, by inputting an incomplete target user image, a two-dimensional key point position, and a target clothing image, a completed virtual fitting image is output.

[0071] Optionally, determining the offset value according to the target clothing image includes:

[0072] Decomposing the target clothing image to obtain a diffuse color texture map, a normal vector texture map, a highlight texture map, and a roughness texture map;

[0073] Inputting the diffuse color texture map, the normal vector texture map, the highlight texture map, and the roughness texture map into an encoder to obtain a plurality of low-dimensional codes;

[0074] averaging the vectors corresponding to the multiple low-dimensional codes to obtain the material information of the target clothing;

[0075] The offset value is generated according to the material information.

[0076] In this embodiment, some of the clothing fabrics we use every day are relatively rough, such as cotton and linen, and some are relatively smooth, such as silk, which has a slight luster under light. Some have special textures, such as the fabric texture of wool in sweaters. This module can decompose the clothing locally to obtain texture maps representing the properties of these materials and apply them to the posture estimation step. Considering that different parts of a piece of clothing may be composed of different materials, semantic segmentation is first performed on different areas of the clothing here, which are simply divided into three areas: sleeves, torso, and collar.

[0077] Image decomposition refers to encoding an image containing clothing into a low-dimensional space, and then decoding the low-dimensional space to obtain a diffuse color texture map, a normal texture map, a highlight texture map, and a roughness texture map corresponding to the image. In this embodiment, an open source material data set is used to train an autoencoder for material estimation. The encoder encodes the input material image into a low-dimensional space, and then decodes the diffuse color texture map, the normal texture map, the highlight texture map, and the roughness texture map from the space. After the training is completed, a specific low-dimensional space encoding can represent a specific material, and the training method of the encoder is supervised learning. Among them, the diffuse color texture map is a texture map used to calculate the lighting and shadow effects. In computer graphics, the diffuse color texture map is usually used to simulate the diffuse reflection properties of the surface of an object, that is, the uniform reflection of light on the surface of the object. This texture map can contain the color information of the surface of the object, helping the rendering engine to calculate the appearance of the object under different lighting conditions. By combining the diffuse color texture map with the lighting model, a more realistic rendering effect can be achieved. The normal texture map is a texture map used to store the normal vector information of the surface of each pixel. In computer graphics, a normal vector is a vector perpendicular to a surface, which is used to describe the orientation of a surface in space. Normal texture maps are often used to provide correct normal vector information for each pixel during the rendering process, so as to accurately simulate the reflection and shadow of the surface when calculating the lighting effect. By combining normal texture maps with diffuse color texture maps, the rendering engine can more accurately simulate the details and lighting effects of the object surface, thereby achieving more realistic rendering results. A specular texture map is a texture map used to simulate the specular reflection properties of the surface of an object. In computer graphics, specular reflection refers to the bright highlights produced when light hits the surface of an object due to the different smoothness or reflectivity of the surface. A specular texture map is often used to specify the intensity and distribution of highlights on the surface of an object, helping the rendering engine to accurately simulate the specular reflection effect of the object during the rendering process. By combining a specular texture map with a diffuse color texture map and a normal texture map, more realistic lighting effects and visual effects can be achieved. A roughness texture map is a texture map used to describe the roughness or smoothness of an object surface. In computer graphics, roughness refers to the degree of irregularity of a surface, that is, the degree to which the surface microstructure scatters light. Roughness textures are often used to specify the roughness properties of an object's surface, helping rendering engines accurately simulate the light reflection characteristics of the object's surface during the rendering process. By adjusting the value of the roughness texture, you can control the smoothness of the object's surface, thereby affecting the appearance of the object under different lighting conditions. Roughness textures are often used together with diffuse color textures, normal vector textures, and highlight textures. Figure 1 Use together to achieve a more realistic rendering effect.

[0078] Use a 128*128 sliding window to slide across different clothing areas in the model image. For each area that is slid across, use the material estimation model described in step 6 to encode it to obtain multiple low-dimensional codes. The encoded vectors are averaged to represent the material of the area. The decoder of the material estimation model decodes and obtains the diffuse color texture map, normal vector texture map, highlight texture map, and roughness texture map corresponding to the material of the area.

[0079] Using non-overlapping sliding windows, the model's clothing image is traversed by sliding with a large step size, thereby calculating the correspondence between each window and the SMPL texture of the target body image, so as to apply the corresponding material properties to the corresponding area and generate an offset value based on the material information.

[0080] Optionally, before performing parameterized 3D human body modeling on the first correspondence and the second correspondence respectively using the target 3D human body model to obtain a first texture image and a second texture image, the method further includes:

[0081] Acquire a training data set, wherein the training data set includes a plurality of human body images;

[0082] Inputting the training data set into a regression model, and outputting camera parameters, motion parameters, and body shape parameters, wherein the camera parameters are parameters of a camera used to shoot the plurality of human body images, and the motion parameters and the body shape parameters are parameters constituting the target three-dimensional human body model;

[0083] The three-dimensional human body model is updated according to the camera parameters, the action parameters and the body shape parameters to obtain the target three-dimensional human body model.

[0084] In this embodiment, before using the target three-dimensional human body model, the three-dimensional human body model needs to be updated and improved to obtain a trained target three-dimensional human body model. Specifically, the parameters of the SMPL model are estimated by training a regression model, wherein the regression model establishes a mathematical function to describe the mapping relationship between the two by analyzing the relationship between the input features and the output targets. Specifically, the regression model is used to predict the two parameters required by the SMPL model from a single input image, namely, the action parameters and the body shape parameters. For the virtual fitting scene, this scheme proposes to use synthetic data and real clothing scene data to train the model. In terms of synthetic data, because SMPL can be used in three-dimensional software, combined with clothing simulation software, a batch of three-dimensional human images with different body shapes and actions wearing common styles of clothes can be synthesized. For these images, different parameters are set for the camera to render, and a batch of human images wearing clothes and their corresponding SMPL parameters are obtained. The model uses these human images as input and regresses to obtain camera parameters and SMPL action parameters and body shape parameters. Among them, camera parameters refer to parameters used to describe the properties and settings of virtual or actual cameras in computer graphics and computer vision. These parameters can affect the camera's viewing angle, projection mode, focal length, perspective effect, etc., thus affecting the final rendered or captured image or video. Action parameters are used in the SMPL model to describe the posture and movement changes of the human body. These parameters can control the joint angles and postures of the human body in three-dimensional space, thereby realizing human motion simulation and animation. Action parameters usually include information such as rotation angle, bending degree and torsion of joints, which are used to generate the posture of the human body under different movements. The body shape parameters of SMPL are used to describe the appearance of the human body. These parameters can control the overall body shape, height, weight, and size and proportion of different parts of the human body. By adjusting the body shape parameters of the SMPL model, the human body shape can be simulated and controlled, thereby generating virtual characters with different body shape characteristics.

[0085] After initial training with synthetic data, fine-tuning is performed using real clothing model images. The two-dimensional projection loss function and adversarial loss function are used during fine-tuning. In the two-dimensional projection loss function, the two-dimensional coordinates of the key points of the human body calculated using the Openpose framework are considered to be accurate two-dimensional coordinates. The estimated key point motion parameters are projected onto a two-dimensional plane based on the estimated camera parameters, and the loss is calculated with the two-dimensional coordinates estimated by Openpose. The adversarial loss is to prevent the network from estimating non-human distorted postures, and a discriminator is used to determine whether the SMPL model output by the network is reasonable.

[0086] Optionally, generating the offset value according to the material information includes:

[0087] Segmenting the target clothing image by a semantic segmentation model to obtain a plurality of clothing regions, wherein different clothing regions correspond to different positions of the target clothing;

[0088] Projecting the target three-dimensional human body model on a target plane to obtain a target vector, wherein the target plane is a camera plane of the target three-dimensional human body model;

[0089] Determine at least one target clothing area among the multiple clothing areas according to the target vector, the target clothing area being a clothing area among the multiple clothing areas that intersects with the target vector;

[0090] The at least one target clothing area is input into the displacement model to obtain the offset value, and the displacement model is used to calculate the displacement offset of the at least one target clothing area on the target user image.

[0091] In this embodiment, the data set established in the above embodiment includes three-dimensional clothes of various common styles, three-dimensional human models of different body shapes and movements, and three-dimensional fitting pictures of any combination thereof. This data set can be used to learn to estimate the position offset of clothes covering the human body under different human postures. Here, only the deformation of clothes caused by different human postures is considered, and different human body shapes are not considered, because the premise considered in this scheme is that the clothes are all fitted. It should be noted that different positions of target clothing corresponding to different clothing areas, such as a certain clothing area corresponding to the sleeve position, another clothing area corresponding to the neckline position, etc., are not specifically limited in this embodiment.

[0092] Specifically, to explain how to use the training dataset, we first need pairs of images of the same piece of clothing in different human body movements. Since not all clothes are in contact with the human body and are squeezed by the human body, such as the hem of a skirt, we first need to segment the parts of the clothes that interact with the human body.

[0093] Secondly, the target clothing image is segmented using a semantic segmentation model to obtain the clothing area to be tried on. Through the estimated target three-dimensional human body model, that is, each pixel point can correspond to a point on the surface of the three-dimensional human body model, and the camera parameters are also estimated, the normal vector of the point is projected onto the target plane, that is, the camera plane, through the camera matrix to obtain the target vector. Exemplarily, the target vector can be a two-dimensional vector. Specifically, the intersection of the coordinates of the two-dimensional direction and the above-segmented clothing area to be tried on is the area where the clothing interacts with the human body. The camera plane refers to the plane used to project the scene in three-dimensional computer graphics, which is usually used to simulate the perspective of the camera or observer. During the rendering process, objects and geometric bodies in the three-dimensional scene will be projected onto the camera plane, and finally a two-dimensional image is formed. The position and direction of the camera plane determine the perspective and content of the final rendered image. The camera plane is usually located at the focus of the camera. According to the parameters and settings of the camera, the position, size and direction of the camera plane can be adjusted to obtain different rendering effects and perspectives.

[0094] By calculating all points on the human body surface, we can get all the areas on the clothes that have an effect on the human body. In this process, the set of pixel points is recorded, and the distance in the two-dimensional direction is the distance from the human body surface. In this embodiment, the displacement model is an offset value calculation formula. Specifically, it is defined that on a certain human body, under the effect of the human body, the distance from the clothing covering the human body to the human body surface is G. Therefore, the displacement model satisfies the following formula

[0095]

[0096] Among them, G Tpose (V, g) is the displacement of the clothing relative to the human body surface when the human body is posed in Tpose. V represents the vertices of the estimated SMPL human body model, and g is the feature vector used to represent the clothing. is the displacement outside T pose caused by human body movement, θ is the coordinates of the key points on the two-dimensional image estimated by openpose as described in step 1, and finally the relative displacement of the clothing relative to the human body surface is obtained by adding them together. Tpose and are two functions that can be obtained based on the above three-dimensional human clothing dataset.

[0097] For the target clothing image, the SMPL human body model and distance G are calculated using the above human body posture estimation method, recorded as V model and G model , and then according to the coordinate θ model, at this time, only g is unknown in the above formula, so the g feature of the clothes shown in the model image can be calculated. After obtaining g, the human body image is estimated to obtain V of the human body image user , use Openpose to estimate θ user ,At this time, you only need to substitute the three variables into the above formula to get the clothing offset on the human body diagram.

[0098] The present application obtains a first correspondence and a second correspondence relative to a target three-dimensional human body model after performing posture estimation on a target user image containing a user and a target clothing image containing a target model wearing target clothing, thereby obtaining a first texture image and a second texture image after modeling, and migrating the second texture image to the first texture image to obtain a target image of the target user wearing the target clothing, thereby improving the effect of virtual fitting.

[0099] See also Figure 3 , Figure 3 is a structural diagram of an image acquisition device provided in an embodiment of the present application. Figure 3 As shown, the image acquisition device 300 includes:

[0100] An estimation module 310 is used to perform posture estimation on the target user image and the target clothing image respectively to obtain a first corresponding relationship and a second corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the target user and the target three-dimensional human body model, and the second corresponding relationship is a corresponding relationship between the target model and the target three-dimensional human body model;

[0101] A modeling module 320, configured to perform parameterized three-dimensional human body modeling on the first correspondence relationship and the second correspondence relationship respectively using the target three-dimensional human body model to obtain a first texture image and a second texture image;

[0102] The generating module 330 is used to transfer the second texture image to the first texture image to generate a target image of the target user wearing the target clothing.

[0103] Optionally, the estimation module 310 includes:

[0104] A first input submodule, used to input the target user image and the target clothing image into a human body two-dimensional posture estimation model, and output a plurality of first feature maps and a plurality of second feature maps, wherein the plurality of first feature maps respectively include a plurality of body parts of the target user, and the plurality of second feature maps respectively include a plurality of body parts of a model corresponding to the target clothing image;

[0105] An estimation submodule is used to perform three-dimensional posture estimation on the multiple first feature maps and the multiple second feature maps respectively based on a target convolutional neural network to obtain the first corresponding relationship and the second corresponding relationship, and the target convolutional neural network is generated based on the target three-dimensional human body model.

[0106] Optionally, the generating module 330 includes:

[0107] A combining submodule is used to fill the pixel values ​​on the second texture image into the standard texture expansion image to obtain a third texture image, wherein the standard texture expansion image is obtained by expanding a standard human body three-dimensional model;

[0108] A migration determination submodule, used to determine an offset value according to a target clothing image, wherein the offset value is used to indicate parameter information during image migration;

[0109] a migration submodule, configured to migrate the third texture image to the first texture image according to the offset value to generate a migration image;

[0110] The second input submodule is used to input the migration image into the conditional diffusion model for image repair and image denoising to obtain the target image.

[0111] Optionally, also include:

[0112] An image decomposition module is used to decompose the target clothing image to obtain a diffuse color texture map, a normal vector texture map, a highlight texture map and a roughness texture map;

[0113] An image input module, used for inputting the diffuse color texture map, the normal vector texture map, the highlight texture map and the roughness texture map into an encoder to obtain a plurality of low-dimensional codes;

[0114] An image calculation module, used for calculating the average of the vectors corresponding to the multiple low-dimensional codes to obtain the material information of the target clothing;

[0115] An image generation module is used to generate the offset value according to the material information.

[0116] Optionally, the image generation module includes:

[0117] An image segmentation submodule, used to segment the target clothing image through a semantic segmentation model to obtain a plurality of clothing regions, where different clothing regions correspond to different positions of the target clothing;

[0118] A projection submodule, used for projecting the target three-dimensional human body model to obtain a target vector, wherein the target plane is a camera plane of the target three-dimensional human body model;

[0119] A determination submodule, configured to determine at least one target clothing area among the multiple clothing areas according to the target vector, wherein the target clothing area is a clothing area among the multiple clothing areas that intersects with the target vector;

[0120] The third input submodule is used to input the at least one target clothing area into the displacement model to obtain the offset value, and the displacement model is used to calculate the displacement offset of the at least one target clothing area on the target user image.

[0121] Optionally, also include:

[0122] An acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of human body images;

[0123] An input module, used to input the training data set into a regression model, and output camera parameters, motion parameters, and body shape parameters, wherein the camera parameters are parameters of a camera used to shoot the plurality of human body images, and the motion parameters and the body shape parameters are parameters constituting the target three-dimensional human body model;

[0124] An updating module is used to update the three-dimensional human body model according to the camera parameters, the action parameters and the body shape parameters to obtain the target three-dimensional human body model.

[0125] The present application obtains a first correspondence and a second correspondence relative to a target three-dimensional human body model after performing posture estimation on a target user image containing a user and a target clothing image containing a target model wearing target clothing, thereby obtaining a first texture image and a second texture image after modeling, and migrating the second texture image to the first texture image to obtain a third image of the target user wearing the target clothing, thereby improving the effect of virtual fitting.

[0126] The present application also provides an electronic device. Figure 4 , the electronic device may include a processor 401, a memory 402, and a program 4021 stored in the memory 402 and executable on the processor 401.

[0127] When the program 4021 is executed by the processor 401, it can achieve Figure 1 Any step in the corresponding method embodiment:

[0128] Performing posture estimation on the target user image and the target clothing image respectively to obtain a first corresponding relationship and a second corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the target user and the target three-dimensional human body model, and the second corresponding relationship is a corresponding relationship between the target model and the target three-dimensional human body model;

[0129] Using the target three-dimensional human body model, respectively, parameterized three-dimensional human body modeling is performed on the first corresponding relationship and the second corresponding relationship to obtain a first texture image and a second texture image;

[0130] The second texture image is transferred to the first texture image to generate a target image of the target user wearing the target clothing.

[0131] Optionally, performing posture estimation on the target user image and the target clothing image respectively to obtain a first corresponding relationship and a second corresponding relationship includes:

[0132] Inputting the target user image and the target clothing image into a human body two-dimensional posture estimation model respectively, and outputting a plurality of first feature maps and a plurality of second feature maps, wherein the plurality of first feature maps respectively include a plurality of body parts of the target user, and the plurality of second feature maps respectively include a plurality of body parts of the model corresponding to the target clothing image;

[0133] Based on the target convolutional neural network, three-dimensional posture estimation is performed on the multiple first feature maps and the multiple second feature maps respectively to obtain the first corresponding relationship and the second corresponding relationship. The target convolutional neural network is generated based on the target three-dimensional human body model.

[0134] Optionally, the step of migrating the second texture image to the first texture image to generate a target image of the target user wearing the target clothing includes:

[0135] Filling the pixel values ​​on the second texture image into a standard texture expansion image to obtain a third texture image, wherein the standard texture expansion image is obtained by expanding a standard human body three-dimensional model;

[0136] Determine an offset value according to the target clothing image, wherein the offset value is used to indicate parameter information during image migration;

[0137] Migrating the third texture image to the first texture image according to the offset value to generate a migrated image;

[0138] The migration image is input into a conditional diffusion model for image repair and image denoising to obtain the target image.

[0139] Optionally, determining the offset value according to the target clothing image includes:

[0140] Decomposing the target clothing image to obtain a diffuse color texture map, a normal vector texture map, a highlight texture map, and a roughness texture map;

[0141] Inputting the diffuse color texture map, the normal vector texture map, the highlight texture map, and the roughness texture map into an encoder to obtain a plurality of low-dimensional codes;

[0142] averaging the vectors corresponding to the multiple low-dimensional codes to obtain the material information of the target clothing;

[0143] The offset value is generated according to the material information.

[0144] Optionally, generating the offset value according to the material information includes:

[0145] Segmenting the target clothing image by a semantic segmentation model to obtain a plurality of clothing regions, wherein different clothing regions correspond to different positions of the target clothing;

[0146] Projecting the target three-dimensional human body model on a target plane to obtain a target vector, wherein the target plane is a camera plane of the target three-dimensional human body model;

[0147] Determine at least one target clothing area among the multiple clothing areas according to the target vector, the target clothing area being a clothing area among the multiple clothing areas that intersects with the target vector;

[0148] The at least one target clothing area is input into the displacement model to obtain the offset value, and the displacement model is used to calculate the displacement offset of the at least one target clothing area on the target user image.

[0149] Optionally, before performing parameterized 3D human body modeling on the first correspondence and the second correspondence respectively using the target 3D human body model to obtain a first texture image and a second texture image, the method further includes:

[0150] Acquire a training data set, wherein the training data set includes a plurality of human body images;

[0151] Inputting the training data set into a regression model, and outputting camera parameters, motion parameters, and body shape parameters, wherein the camera parameters are parameters of a camera used to shoot the plurality of human body images, and the motion parameters and the body shape parameters are parameters constituting the target three-dimensional human body model;

[0152] The three-dimensional human body model is updated according to the camera parameters, the action parameters and the body shape parameters to obtain the target three-dimensional human body model.

[0153] The present application obtains a first correspondence and a second correspondence relative to a target three-dimensional human body model after performing posture estimation on a target user image containing a user and a target clothing image containing a target model wearing target clothing, thereby obtaining a first texture image and a second texture image after modeling, and migrating the second texture image to the first texture image to obtain a third image of the target user wearing the target clothing, thereby improving the effect of virtual fitting.

[0154] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned method for acquiring images can be implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0155] An embodiment of the present application further provides a computer program product, which is stored in a storage medium. The computer program product is executed by at least one processor to implement the various processes of the above-mentioned image acquisition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0156] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0158] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for acquiring an image, It is characterized in that The method comprises: Performing posture estimation on the target user image and the target clothing image respectively to obtain a first corresponding relationship and a second corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the target user and the target three-dimensional human body model, and the second corresponding relationship is a corresponding relationship between the target model and the target three-dimensional human body model; Using the target three-dimensional human body model, respectively, parameterized three-dimensional human body modeling is performed on the first corresponding relationship and the second corresponding relationship to obtain a first texture image and a second texture image; The second texture image is transferred to the first texture image to generate a target image of the target user wearing the target clothing.

2. The method according to claim 1, It is characterized in that The step of respectively estimating the posture of the target user image and the target clothing image to obtain a first corresponding relationship and a second corresponding relationship includes: Inputting the target user image and the target clothing image into a human body two-dimensional posture estimation model respectively, and outputting a plurality of first feature maps and a plurality of second feature maps, wherein the plurality of first feature maps respectively include a plurality of body parts of the target user, and the plurality of second feature maps respectively include a plurality of body parts of the model corresponding to the target clothing image; Based on the target convolutional neural network, three-dimensional posture estimation is performed on the multiple first feature maps and the multiple second feature maps respectively to obtain the first corresponding relationship and the second corresponding relationship. The target convolutional neural network is generated based on the target three-dimensional human body model.

3. The method according to claim 1, It is characterized in that The step of migrating the second texture image to the first texture image to generate a target image of the target user wearing the target clothing includes: Filling the pixel values ​​on the second texture image into a standard texture expansion image to obtain a third texture image, wherein the standard texture expansion image is obtained by expanding a standard human body three-dimensional model; Determine an offset value according to the target clothing image, wherein the offset value is used to indicate parameter information during image migration; Migrating the third texture image to the first texture image according to the offset value to generate a migrated image; The migration image is input into a conditional diffusion model for image repair and image denoising to obtain the target image.

4. The method according to claim 3, It is characterized in that The step of determining the offset value according to the target clothing image includes: Decomposing the target clothing image to obtain a diffuse color texture map, a normal vector texture map, a highlight texture map, and a roughness texture map; Inputting the diffuse color texture map, the normal vector texture map, the highlight texture map, and the roughness texture map into an encoder to obtain a plurality of low-dimensional codes; averaging the vectors corresponding to the multiple low-dimensional codes to obtain the material information of the target clothing; The offset value is generated according to the material information.

5. The method according to claim 4, It is characterized in that The generating the offset value according to the material information includes: Segmenting the target clothing image by a semantic segmentation model to obtain a plurality of clothing regions, wherein different clothing regions correspond to different positions of the target clothing; Projecting the target three-dimensional human body model on a target plane to obtain a target vector, wherein the target plane is a camera plane of the target three-dimensional human body model; Determine at least one target clothing area among the multiple clothing areas according to the target vector, the target clothing area being a clothing area among the multiple clothing areas that intersects with the target vector; The at least one target clothing area is input into the displacement model to obtain the offset value, and the displacement model is used to calculate the displacement offset of the at least one target clothing area on the target user image.

6. The method according to claim 1, It is characterized in that Before performing parameterized 3D human body modeling on the first correspondence and the second correspondence using the target 3D human body model to obtain the first texture image and the second texture image, the method further includes: Acquire a training data set, wherein the training data set includes a plurality of human body images; Inputting the training data set into a regression model, and outputting camera parameters, motion parameters, and body shape parameters, wherein the camera parameters are parameters of a camera used to shoot the plurality of human body images, and the motion parameters and the body shape parameters are parameters constituting the target three-dimensional human body model; The three-dimensional human body model is updated according to the camera parameters, the action parameters and the body shape parameters to obtain the target three-dimensional human body model.

7. An image acquisition device, It is characterized in that The device comprises: an estimation module, configured to perform posture estimation on the target user image and the target clothing image respectively, to obtain a first corresponding relationship and a second corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the target user and the target three-dimensional human body model, and the second corresponding relationship is a corresponding relationship between the target model and the target three-dimensional human body model; A modeling module, used for performing parameterized three-dimensional human body modeling on the first corresponding relationship and the second corresponding relationship respectively by using the target three-dimensional human body model to obtain a first texture image and a second texture image; A generating module is used to transfer the second texture image to the first texture image to generate a target image of the target user wearing the target clothing.

8. An electronic device, include: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the image acquisition method as described in any one of claims 1 to 6.

9. A readable storage medium for storing a program, It is characterized in that When the program is executed by a processor, the steps in the image acquisition method according to any one of claims 1 to 6 are implemented.

10. A computer program product, It is characterized in that The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps in the image acquisition method according to any one of claims 1 to 6.

Citation Information

Cited By

  • Model generation method, storage medium and program product

    CN122435103A