Virtual character graph generation method and device, equipment, medium and product

Through segmentation and mapping technology, combined with lighting processing, the problem of low deformation and fit of clothes in virtual fittings is solved, and the display effect and realism of virtual character pictures are improved.

CN119027586APending Publication Date: 2024-11-26MIGU CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411070396.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Among the existing virtual fitting technology, the clothes are seriously deformed, have low fit with humans, and have poor effect on complex structures, resulting in the virtual fitting technology not being widely used.

Method used

The pre-trained clothing area segmentation model and model area segmentation model are used to divide the clothing diagram and character diagram into different areas. The area to be deformed is obtained through union calculations, and the clothing area is mapped to the human body area, combining the lighting decomposition model and confrontation generation Network, optimize the generation of virtual character maps.

Benefits of technology

It reduces the deformation of clothes in the virtual character picture, improves the fit between the clothes and the human body, improves the display effect, and enhances the reality of the virtual character picture.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027586A_ABST
    Figure CN119027586A_ABST
Patent Text Reader

Abstract

The invention discloses a virtual character graph generation method and apparatus, a device, a medium and a product. The method comprises the steps of segmenting an input garment graph into garment regions according to a pre-trained garment region segmentation model; according to a pre-trained model region segmentation model, segmenting the input figure graph into human body regions; a union set of the human body areas and the corresponding clothes areas is calculated, and areas to be deformed are obtained respectively; and mapping the clothing area in each to-be-deformed area to a corresponding human body area to generate a virtual character graph. According to the scheme of the invention, through the clothes migration method for performing regional deformation on different parts, deformation processing is performed on each local image, so that clothes deformation in the virtual character graph can be reduced, the fitting degree of the clothes and the body of the model is improved, and the display effect is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment, medium and product for generating a virtual character image. Background Art

[0002] With the development of AIGC applications and the metaverse, the industry has applied virtual synthesis technology to various aspects that require the display of virtual human images, and has produced research on virtual fitting using computers. However, there are still various practical application problems in the application scenarios of virtual fitting and virtual characters. For example, clothes are severely deformed, the fit between clothes and the human body is not high, and the effect of complex structured clothing is poor. The existence of these problems has led to the fact that virtual fitting technology has not been widely used. Summary of the invention

[0003] In order to solve the above problems, the present invention proposes a method, device, equipment, medium and product for generating a virtual character image, which can reduce the deformation of clothes in the virtual character image, improve the fit between the clothes and the model's body, and enhance the display effect.

[0004] An embodiment of the present invention provides a method for generating a virtual character image, comprising:

[0005] Segment the input clothing image into various clothing regions according to the pre-trained clothing region segmentation model;

[0006] The input character image is segmented into various human body regions according to the pre-trained model region segmentation model;

[0007] Calculate the union set of each human body region and the corresponding clothing region to obtain each region to be deformed;

[0008] The clothing areas in the respective areas to be deformed are mapped to corresponding human body areas to generate a virtual character map.

[0009] Preferably, the method further comprises:

[0010] Extracting illumination parameters from the character image according to a pre-trained illumination decomposition model;

[0011] A diffuse reflection formula is constructed according to the extracted illumination parameters, and the virtual character image is calculated to obtain the virtual character image after illumination reconstruction.

[0012] Preferably, mapping the clothing regions in the regions to be deformed to corresponding human body regions comprises:

[0013] In the clothing area of ​​each to-be-deformed area and the corresponding human body area, N control points are uniformly sampled along the edge of the image;

[0014] Inputting N control points sampled from the clothing area of ​​each to-be-deformed area and N control points sampled from the corresponding human body area into a preset spline function for solving, thereby obtaining the parameters to be solved in the spline function;

[0015] Inputting the desired parameter into the spline function to obtain a mapping function;

[0016] According to the mapping function, the pixel points on the clothing area of ​​each to-be-deformed area are mapped to the corresponding human body area to obtain a mapped image.

[0017] Preferably, generating a virtual character map includes:

[0018] The images mapped in different areas to be deformed are stitched together to obtain the initial effect image;

[0019] Using the Openpose framework to estimate the positions of key points of the character image, and obtain the positions of key points of the human body;

[0020] Using a Densepose model to perform dense semantic segmentation of the human body on the character image, one-to-one correspondence between the positions of the key points of the human body and the face elements on the 3D human body model, spatial transformation of the coordinates of the key points of the human body, and obtaining UV coordinates of the key points of the human body as human body posture information;

[0021] The initial effect image and the human body posture information are input into a preset adversarial generation network for optimization, and the virtual character image is output.

[0022] Preferably, the illumination decomposition model training process specifically includes:

[0023] Obtain illumination images of the same 3D human body model under different illumination parameters as a synthetic data set;

[0024] Inputting the synthetic data set and the corresponding illumination parameters into a preset illumination information extraction network for learning in a self-supervised manner, and respectively training the diffuse reflection estimation branch, the illumination estimation branch, and the normal vector branch in the illumination information extraction network using the corresponding illumination parameters to obtain the illumination decomposition model;

[0025] The lighting parameters include lighting diffuse reflection color, lighting normal vector and lighting spherical harmonic coefficients.

[0026] Preferably, the clothing region segmentation model or the model region segmentation model training process specifically includes:

[0027] Obtain image data from open source datasets and annotate them to obtain training datasets;

[0028] Inputting the training data set into a preset semantic segmentation network for semantic segmentation training to obtain a preliminary training model;

[0029] Obtain image data in multiple scenes for annotation as the second training data set;

[0030] The second training data set is input into the preliminary training model for domain adaptation training to obtain the clothing region segmentation model or the model region segmentation model.

[0031] An embodiment of the present invention provides a device for generating a virtual character image, the device comprising:

[0032] A clothing segmentation module is used to segment the input clothing image into various clothing regions according to a pre-trained clothing region segmentation model;

[0033] A human body segmentation module is used to segment the input human image into various human body regions according to a pre-trained model region segmentation model;

[0034] A region calculation module, used to calculate the union set of each human body region and the corresponding clothing region, and obtain each region to be deformed;

[0035] The mapping module is used to map the clothing areas in each of the areas to be deformed to the corresponding human body areas to generate a virtual character map.

[0036] Preferably, the device further comprises an illumination module, which is used for:

[0037] Extracting illumination parameters from the character image according to a pre-trained illumination decomposition model;

[0038] A diffuse reflection formula is constructed according to the extracted illumination parameters, and the virtual character image is calculated to obtain the virtual character image after illumination reconstruction.

[0039] Preferably, the mapping module is specifically used for:

[0040] In the clothing area of ​​each to-be-deformed area and the corresponding human body area, N control points are uniformly sampled along the edge of the image;

[0041] Inputting N control points sampled from the clothing area of ​​each to-be-deformed area and N control points sampled from the corresponding human body area into a preset spline function for solving, thereby obtaining the parameters to be solved in the spline function;

[0042] Inputting the desired parameter into the spline function to obtain a mapping function;

[0043] According to the mapping function, the pixel points on the clothing area of ​​each to-be-deformed area are mapped to the corresponding human body area to obtain a mapped image.

[0044] Preferably, the mapping module is specifically used for:

[0045] Splicing the images mapped from the regions to be deformed to obtain an initial rendering;

[0046] Using the Openpose framework to estimate the positions of key points of the character image, and obtain the positions of key points of the human body;

[0047] Using a Densepose model to perform dense semantic segmentation of the human body on the character image, one-to-one correspondence between the positions of the key points of the human body and the face elements on the 3D human body model, spatial transformation of the coordinates of the key points of the human body, and obtaining UV coordinates of the key points of the human body as human body posture information;

[0048] The initial effect image and the human body posture information are input into a preset adversarial generation network for optimization, and the virtual character image is output.

[0049] Preferably, the illumination decomposition model training process specifically includes:

[0050] Obtain illumination images of the same 3D human body model under different illumination parameters as a synthetic data set;

[0051] Inputting the synthetic data set and the corresponding illumination parameters into a preset illumination information extraction network for learning in a self-supervised manner, and respectively training the diffuse reflection estimation branch, the illumination estimation branch, and the normal vector branch in the illumination information extraction network using the corresponding illumination parameters to obtain the illumination decomposition model;

[0052] The lighting parameters include lighting diffuse reflection color, lighting normal vector and lighting spherical harmonic coefficients.

[0053] Preferably, the clothing region segmentation model or the model region segmentation model training process specifically includes:

[0054] Obtain image data from open source datasets and annotate them to obtain training datasets;

[0055] Inputting the training data set into a preset semantic segmentation network for semantic segmentation training to obtain a preliminary training model;

[0056] Obtain image data in multiple scenes for annotation as the second training data set;

[0057] The second training data set is input into the preliminary training model for domain adaptation training to obtain the clothing region segmentation model or the model region segmentation model.

[0058] An embodiment of the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a method for generating a virtual character image as described in any of the above embodiments is implemented.

[0059] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a method for generating a virtual character image as described in any of the above embodiments.

[0060] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0061] Compared with the prior art, the present invention provides a method, device, equipment, medium and product for generating a virtual character image, which divides an input clothing image into various clothing areas according to a pre-trained clothing area segmentation model; divides an input character image into various human body areas according to a pre-trained model area segmentation model; calculates the union set of each human body area and the corresponding clothing area to obtain each area to be deformed; maps the clothing area in each area to be deformed to the corresponding human body area to generate a virtual character image. The present application scheme uses a clothing migration method that performs regional deformation on different parts, and deforms each local image separately, which can reduce the deformation of clothes in the virtual character image, improve the fit between the clothes and the model's body, and enhance the display effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flowchart of a method for generating a virtual character image provided by an embodiment of the present invention;

[0063] Figure 2 is a flowchart of the illumination decomposition model training process provided by an embodiment of the present invention;

[0064] Figure 3 is another flowchart of the illumination decomposition model training process provided by an embodiment of the present invention;

[0065] Figure 4 It is a structural schematic diagram of a virtual character image generating device provided by an embodiment of the present invention;

[0066] Figure 5It is a structural diagram of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] It can be understood that the various embodiments described below can be combined or combined in a logical manner, and the embodiments of the present invention will not describe various combinations one by one.

[0069] With the development of AIGC applications and the metaverse, the industry has applied virtual synthesis technology to various aspects that require the display of virtual human images, such as virtual anchors, virtual characters, etc., but there are still various practical application problems in these application scenarios. For example, in terms of virtual anchors, the current virtual anchors' assets are all pre-made, and the clothing accessories are fixed, but the audience will have aesthetic fatigue of the anchors after watching for a long time. If the anchors can change their clothes over time or with the commentary scenes, such as using formal clothes when hosting large-scale commercial activities, and using sportswear when hosting sports activities, they can not only maintain freshness, but also better adapt to the anchor scenes and further maintain user stickiness. Such tasks involve virtual fitting and virtual characters. There are many studies on such tasks in the industry, but there are still many problems, such as serious deformation of clothes, unrealistic virtual character images, clothes that do not fit the human body, poor effects of complex structured clothing, and mismatch between lighting and the scene where the model is located. These problems need to be solved, so they have not been widely used.

[0070] In view of the above technical problems, this application proposes a method for generating a virtual character map, see Figure 1 , is a flow chart of a method for generating a virtual character image provided by an embodiment of the present invention, the method comprising the following steps:

[0071] Step S1, segmenting the input clothing image into various clothing regions according to a pre-trained clothing region segmentation model;

[0072] Step S2, segmenting the input character image into various human body regions according to a pre-trained model region segmentation model;

[0073] Step S3, calculating the union of each human body region and the corresponding clothing region to obtain each region to be deformed;

[0074] Step S4, mapping the clothing areas in the areas to be deformed to corresponding human body areas to generate a virtual character map.

[0075] During the specific implementation of this embodiment, the user inputs a character image and a clothing image, and a corresponding virtual character image needs to be generated for the character image according to the clothing image.

[0076] In the existing virtual fitting methods, clothes are deformed as a whole, which easily leads to severe deformation of local areas in order to reduce the overall deviation. Therefore, this case divides the input clothing image into different clothing areas according to the pre-trained clothing area segmentation model;

[0077] The input character image is segmented into different human body regions according to the pre-trained model region segmentation model;

[0078] It should be noted that for clothing images, they are divided into different semantic areas such as left and right sleeves, torso, left and right trouser legs, jackets, etc. according to the composition of the clothing. For person images, they are divided into background, the above-mentioned clothing partitions, and human skin and other areas.

[0079] Use the trained semantic segmentation network to predict the input clothing and character images. The two semantic segmentation networks are denoted as Segment and g and Seg p , the input clothing image is denoted as I g , the character is denoted as I p , then:

[0080] l g =Seg g (I g );

[0081] l p =Seg p (I p );

[0082] The clothing image is preprocessed according to semantics. According to the predicted label of each pixel in the image, the masks of different parts of the clothing are obtained, and these masks are used to extract the regional images of different parts of the clothing from the image, that is, in each regional image, except for the area of ​​interest, other areas are covered.

[0083] The clothing image of a top is segmented and labeled into four regions: the top torso, the left sleeve, the right sleeve, and the background. Without considering the background, three region images can be obtained from the original image based on the obtained annotations.

[0084] According to the pre-set correspondence table between clothing areas and human body areas, different human body areas and corresponding clothing areas are taken as the union to obtain different areas to be deformed; for example, taking the left sleeve area of ​​the clothing image as an example, in the figure image, the model's left arm area is found, and the union of the left arm area and the left sleeve area in the figure image is calculated as the area to be deformed of the left arm.

[0085] When the area to be deformed is determined, it can be intercepted with reference to the length information of the target clothing, etc., as the area corresponding to the deformation of the clothing.

[0086] For the clothing partition map and the model partition map in the deformation area, a two-dimensional deformation algorithm of thin plate spline is used to perform deformation operation on each partition, and the clothing areas in different deformation areas are mapped to the corresponding human body areas;

[0087] Then the images are pieced together, and finally the preliminary pieced together images are redrawn using a generative network to obtain a synthesized virtual character image.

[0088] In the virtual fitting scene, the virtual character image obtained by calculating the clothing image and the character image is used as the virtual model image. Users can intuitively see the actual effect of the character wearing clothes through the virtual character image, which is convenient for users to choose clothes.

[0089] This application scheme aims at the problem that clothes do not fit well, are severely deformed, and complex structures cannot be restored when they are transferred to a human body in a given character image. This scheme proposes a method for transferring clothes with regional deformation, which deforms each local image of different parts separately, avoiding the problem of severe deformation of local areas when the clothes are transferred as a whole in the prior art, and can reduce the problems of clothes deformation, poor fit with the human body, and poor display effect in the virtual character image. It can reduce the deformation of clothes in the virtual character image, improve the fit between the clothes and the model's body, and enhance the display effect.

[0090] In another embodiment provided by the present invention, in the existing virtual fitting method, both the data set and the model design are collected and designed based on the character pictures under studio conditions. In actual applications, in order to create a specific atmosphere, the model is often in a more complex lighting environment. At this time, the migrated clothes do not match the lighting of the given character picture, which will lead to a lack of realism.

[0091] In view of the above-mentioned defects of the prior art, after mapping the clothing areas in different areas to be deformed to the corresponding human body areas and generating a virtual character map, the method of the present application further includes:

[0092] The lighting parameters of the character image are extracted according to the pre-trained lighting decomposition model, that is, a single input human image is decomposed according to the rendering equation, the lighting information in the image is estimated, and the lighting parameters are determined. It should be noted that the lighting parameters can simulate the lighting scene on the character image.

[0093] Therefore, a diffuse reflection formula can be constructed according to the extracted illumination parameters, and the virtual character image can be calculated to obtain the virtual character image after illumination reconstruction.

[0094] The present application scheme takes into account the lighting in the original character image, solves the problem of lighting realism in the prior art, further improves the realism of the generated image, and obtains a highly realistic virtual character image.

[0095] In another embodiment provided by the present invention, when mapping clothing regions of different to-be-deformed regions to human body regions, the following steps are specifically performed:

[0096] In the clothing area and the corresponding human body area of ​​different deformation areas, N control points are uniformly sampled along the edge of the image, which are denoted as G N and P N .

[0097] The purpose of the deformation algorithm is to make the source image, that is, the points in the clothing partition map, fall into the target image after deformation calculation.

[0098] The N control points sampled on the clothing area and the corresponding N control points sampled on the human body area are input into the preset spline function for solution, that is, the corresponding points in the model partition map, and the image plane is regarded as a flexible and deformable rigid body. For a two-dimensional image, the spline function in one-dimensional direction (x or y) is as follows:

[0099]

[0100] Among them, (x, y) is the position coordinate of the corresponding control point in the human body area, (x i ,y i ) is the position coordinate of the i-th control point on the clothing area, U(·) is the radial kernel function, and w i 、a1、a x and a y All are parameters to be requested.

[0101] The control points of the N human body regions are taken as the pixels (x, y) of the target image, and the control points of the N clothing regions are taken as the pixels (x i ,y i) is input into the spline function to form a system of equations. According to the constraints of the spline interpolation, the parameters to be determined in the formula can be solved. After obtaining the parameters to be determined, the parameters to be determined are input into the mapping function obtained as the spline function. The value position of each pixel point (x, y) in the target image in the source image can be calculated. According to the obtained mapping relationship, the mapping function is obtained.

[0102] According to the mapping function, the deformed image of each partition image can be calculated. By reassembling these deformed partition images through segmentation information, the initial wearing effect image of the target clothing on the target person image can be obtained.

[0103] In another embodiment provided by the present invention, when generating a complete virtual character image, since the previous region segmentation is only obtained by specifying rules for human body semantic regions and clothing semantic regions, there may be errors in the regions, and the collision interaction between clothing and the human body is not considered, the initial wearing image will look unrealistic. Therefore, after obtaining the initial wearing effect image, it is necessary to input it and the key point network (as human body posture information) obtained in the human body posture estimation module into a generation network for further optimization.

[0104] That is, it is necessary to use a preset semantic model to perform semantic analysis on the character image, determine the human body posture information, and splice the human body posture information and the images mapped to different areas to be deformed to obtain the initial effect image, which is input into the preset adversarial network for optimization, and the virtual character image is output to optimize the collision relationship between the human body and clothing.

[0105] Construct a semantic model to estimate the human posture of the character image, obtain the positions of human body parts and key points of the human body, and use the Openpose framework to estimate the positions of key points of the character image. Openpose is a real-time two-dimensional human posture estimation model based on a convolutional deep network. It extracts features from the input image containing the human body through a convolutional network to obtain a set of feature maps, and then uses two branches to extract the part position information map and the part affinity map, detects the positions of different parts of the human body and connects the parts to obtain a complete human skeleton and the positions of pre-defined human key points. In this step, input the image containing the human body into Openpose, and the model will output the estimated positions of 25 key points of the human body in the character image. Etc., to provide the model's human body position information for synthesizing virtual character images.

[0106] Use the Densepose model to perform dense semantic segmentation of the human body in the character image. Densepose is also a model for human posture estimation, but unlike Openpose, which estimates the position of key points, Densepose estimates dense annotations. Densepose uses a convolutional deep network with pyramid network features to match the surface pixels of the human body in the input 2D image with the face elements on the 3D human body model one by one, and transforms the predicted spatial coordinates of the human body surface to obtain UV coordinates and record them on the corresponding pixels of the two-dimensional image. At the same time, it can also complete the semantic segmentation of the human body. Each pixel in the image will record the UV coordinates and the body part label to which it belongs.

[0107] Then, the initial effect image and the human body posture information are input into a preset adversarial generation network for optimization, and the virtual character image is output.

[0108] The adversarial generative network is trained using an adversarial loss function and an open source VITON dataset.

[0109] The initial renderings are analyzed for collision based on human body posture information to improve the wearing effect of the clothing.

[0110] In another embodiment provided by the present invention, the purpose of the illumination decomposition model is to train a deep network to decompose a single input human body image according to a rendering equation, and estimate the illumination information, human body surface diffuse reflection information, and normal vector information in the image, so that after the model is dressed, the illumination of the original character image can be reflected on the new clothes, thereby increasing the realism of the virtual character image.

[0111] In this method, the training of this network is defined as a relighting task, that is, an image and a new lighting are input, and the network should input the image of the subject of the image under the new lighting. Because it is difficult to take images of the same subject but different lighting in reality, synthetic data will be used when training this network. Synthetic data will be used when training the lighting decomposition model. The synthetic data set is obtained from a 3D scanned human model wearing clothes. By rendering the same 3D human model with different lighting in the rendering engine, images of the same subject under different lighting can be obtained.

[0112] See also Figure 2 , is a flow chart of the illumination decomposition model training process provided by an embodiment of the present invention. During the training, a synthetic data set is obtained.

[0113] The synthetic data set and the corresponding illumination parameters are input into a preset illumination information extraction network for learning in a self-supervised manner. The synthetic data set and the corresponding illumination parameters are input into a preset illumination information extraction network for learning in a self-supervised manner.

[0114] According to the Lambert diffuse reflection formula, and using spherical harmonics to represent all the light received by a certain point on the object, when the emission direction remains unchanged (applicable to the case of synthesizing pictures in this method, because this method only needs to change the clothing in the figure picture, but does not need to change the shooting angle of the figure picture), the color of a certain point x in the image is the light emitted from the surface of the object corresponding to the pixel point recorded by the camera sensor, which can be expressed as

[0115] Among them, L o (x) refers to the light emitted by the point x, Albedo(x) is the diffuse color of the point, Y is the spherical harmonic function, n(x) is the normal vector of the surface of the point, and L is the coefficient of the spherical harmonic illumination of the scene. The symbol represents the Hadamard product, which is to multiply the elements of the corresponding positions of the two vectors. According to this formula, if we can input the image for decomposition and obtain the three variables Albedo(x), n(x) and L, we can reconstruct the original image according to the formula, and change the variables to synthesize a new image.

[0116] The illumination information extraction network corresponds one-to-one to the three illumination variables of the diffuse reflection formula: illumination diffuse reflection color, illumination normal vector, and illumination spherical harmonic coefficient. It should include three branches: diffuse reflection estimation branch albedo, illumination estimation branch light, and normal vector branch n. Considering that the reflective properties of human skin and clothing are different, in order to allow the network to learn this feature, the output of the diffuse reflection branch will include two parts: albedo and human skin mask.

[0117] When performing self-supervised learning, the illumination parameters including the illumination diffuse color, illumination normal vector and illumination spherical harmonic coefficients are determined by calculating the albedo loss of the diffuse estimation branch, the segmentation loss of the skin mask, the light loss of the illumination estimation branch and the normal loss of the normal vector branch n, and then the diffuse reflection formula of the new illumination in the virtual character image is determined, the re-lighting image is calculated, the loss L2 loss is calculated, the three branches are trained, and the illumination decomposition model is obtained.

[0118] As a preferred solution, the network trained with synthetic data is used here. There is a difference between the learned data distribution and the real data distribution. In order to solve the problem of domain adaptation, a network is added on the basis of the above network to optimize the prediction quality, which is called the refinement network.

[0119] See also Figure 3 , is another flowchart of the illumination decomposition model training process provided by an embodiment of the present invention. For the refinement network, this method uses a self-supervised method for training, uses the collected real person image, inputs it into the image illumination extraction network that has been trained using synthetic data, decomposes albedo, light and n, and uses the diffuse reflection formula to calculate and reconstruct the input image, and inputs the reconstructed image into the refinement network, hoping that the output image after network optimization is as close as possible to the original image, so as to solve the problem of poor domain adaptability caused by differences during model training.

[0120] In another embodiment of the present invention, the clothing region segmentation model or the model region segmentation model uses an end-to-end neural network model, and the neural network is trained with a large amount of labeled data, so that the network can semantically segment the input clothing image, classify each pixel point of the input image, and mark which area of ​​the clothing the pixel point belongs to. During the specific training:

[0121] Training the clothing annotation network. First, define the clothing categories. In this solution, clothing is divided into five categories: tops, pants, skirts, coats, and dresses. Then, define the annotations of different parts according to the categories. The network training is divided into two stages. In the first stage, the open source VITON (virtual try-on) dataset in the industry is used to train the network. According to the correspondence between the image areas and labels defined in Table 1, the clothing images and character images in the dataset are annotated.

[0122] Table 1 Correspondence table between images and regions

[0123] area Label Background or inside of clothes 0 Human skin 1 Top torso 2 Dress Torso 3 Jacket Torso 4 Left sleeve 5 Right sleeve 6 Left trouser leg 7 Right trouser leg 8 Skirt 9 Jacket collar 10 Jacket collar 11

[0124] The annotations are stored in the form of single-channel images. After the annotations are completed, a paired (image-annotation) dataset is obtained. Since there is a significant difference between the distribution of person images and clothing images, in this module, two semantic segmentation networks need to be trained using the person image data pairs and clothing image data pairs respectively, one for the semantic segmentation of person images and the other for the semantic segmentation of clothing images.

[0125] During training, the input of the network is the image, and the output is the prediction of the pixel classification. Since the character images in the VITON dataset are all "clean" character images in the studio style, that is, there is no complex background, no complex lighting, and the model has no large-scale distorted movements, etc., and the character images in daily applications, such as the seller show pictures on major domestic e-commerce platforms, mostly have complex backgrounds, complex lighting, and even outdoor snapshots, so after the first stage of training, the data distribution learned by the network is still a certain distance from the data distribution in real applications. In order to solve this domain difference problem, the second stage of training is required, that is, a batch of more daily character image data is collected for annotation, and the second stage of domain adaptation training is performed on the network that already has better prior knowledge to obtain the final semantic segmentation network. This two-stage training not only makes the network suitable for real character images, but also reduces the manpower cost of data set collection and annotation. The neural networks used in this module are all autoencoder networks with UNet network structure as the main body, which is a commonly used network structure in image semantic segmentation tasks. In terms of training, multi-classification cross entropy loss function is used for training.

[0126] The trained clothing region segmentation model and the model region segmentation model are segmented into different semantic regions such as the left and right sleeves, torso, left and right trouser legs, and jackets. For the character image, they are segmented into the background, the above-mentioned clothing partitions, and human skin regions. The clothing in each region is deformed locally, and finally spliced ​​and further denoised and corrected through a generative network to enhance the realism of the clothing migration effect.

[0127] See also Figure 4 , is a schematic diagram of the structure of a virtual character image generation device provided by an embodiment of the present invention, the device comprising:

[0128] A clothing segmentation module is used to segment the input clothing image into various clothing regions according to a pre-trained clothing region segmentation model;

[0129] A human body segmentation module is used to segment the input human image into various human body regions according to a pre-trained model region segmentation model;

[0130] A region calculation module, used to calculate the union set of each human body region and the corresponding clothing region to obtain each region to be deformed;

[0131] The mapping module is used to map the clothing areas in each of the areas to be deformed to the corresponding human body areas to generate a virtual character map.

[0132] The virtual character image generation device provided in this embodiment can execute all the steps and functions of the virtual character image generation method provided in any of the above embodiments, and the specific functions of the device will not be described in detail here.

[0133] See also Figure 5 , is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. The terminal device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a virtual character image generation program. When the processor executes the computer program, the steps in each of the above-mentioned virtual character image generation method embodiments are implemented, such as Figure 1 Alternatively, the processor implements the functions of each module in the above-mentioned device embodiments when executing the computer program.

[0134] Exemplarily, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the virtual character image generation device. For example, the computer program may be divided into several modules, and the specific functions of each module have been described in detail in a virtual character image generation method provided in any of the above embodiments, and the specific functions of the device will not be repeated here.

[0135] The device for generating a virtual character image may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The device for generating a virtual character image may include, but not limited to, a processor and a memory. A person skilled in the art may understand that the schematic diagram is only an example of a device for generating a virtual character image, and does not constitute a limitation on a device for generating a virtual character image. The device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the device for generating a virtual character image may also include input and output devices, network access devices, buses, etc.

[0136] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the virtual character image generation device, and uses various interfaces and lines to connect the various parts of the entire virtual character image generation device.

[0137] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the virtual character image generation device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0138] Wherein, if the module integrated in the device for generating a virtual character image is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0139] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of any method described in the above embodiments when executed by a processor.

[0140] It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating a virtual character image, characterized in that: The method comprises: Segment the input clothing image into various clothing regions according to the pre-trained clothing region segmentation model; The input character image is segmented into various human body regions according to the pre-trained model region segmentation model; Calculate the union set of each human body region and the corresponding clothing region to obtain each region to be deformed; The clothing areas in the respective areas to be deformed are mapped to corresponding human body areas to generate a virtual character map.

2. The method for generating a virtual character image according to claim 1, characterized in that: The method further comprises: Extracting illumination parameters from the character image according to a pre-trained illumination decomposition model; A diffuse reflection formula is constructed according to the extracted illumination parameters, and the virtual character image is calculated to obtain the virtual character image after illumination reconstruction.

3. The method for generating a virtual character image according to claim 1, characterized in that: Mapping the clothing regions in the regions to be deformed to corresponding human body regions includes: In the clothing area of ​​each to-be-deformed area and the corresponding human body area, N control points are uniformly sampled along the edge of the image; Inputting N control points sampled from the clothing area of ​​each to-be-deformed area and N control points sampled from the corresponding human body area into a preset spline function for solving, thereby obtaining the parameters to be solved in the spline function; Inputting the desired parameter into the spline function to obtain a mapping function; According to the mapping function, the pixel points on the clothing area of ​​each to-be-deformed area are mapped to the corresponding human body area to obtain a mapped image.

4. The method for generating a virtual character image according to claim 1, characterized in that: The generating of the virtual character map comprises: Splicing the images mapped from the regions to be deformed to obtain an initial rendering; Using the Openpose framework to estimate the positions of key points of the character image, and obtain the positions of key points of the human body; Using a Densepose model to perform dense semantic segmentation of the human body on the character image, one-to-one correspondence between the positions of the key points of the human body and the face elements on the 3D human body model, spatial transformation of the coordinates of the key points of the human body, and obtaining UV coordinates of the key points of the human body as human body posture information; The initial effect image and the human body posture information are input into a preset adversarial generation network for optimization, and the virtual character image is output.

5. The method for generating a virtual character image according to claim 2, characterized in that: The illumination decomposition model training process specifically includes: Obtain illumination images of the same 3D human body model under different illumination parameters as a synthetic data set; Inputting the synthetic data set and the corresponding illumination parameters into a preset illumination information extraction network for learning in a self-supervised manner, and respectively training the diffuse reflection estimation branch, the illumination estimation branch, and the normal vector branch in the illumination information extraction network using the corresponding illumination parameters to obtain the illumination decomposition model; The lighting parameters include lighting diffuse reflection color, lighting normal vector and lighting spherical harmonic coefficients.

6. The method for generating a virtual character image according to claim 1, characterized in that: The clothing region segmentation model or the model region segmentation model training process specifically includes: Obtain image data from open source datasets and annotate them to obtain training datasets; Inputting the training data set into a preset semantic segmentation network for semantic segmentation training to obtain a preliminary training model; Obtain image data in multiple scenes for annotation as the second training data set; The second training data set is input into the preliminary training model for domain adaptation training to obtain the clothing region segmentation model or the model region segmentation model.

7. A virtual character image generating device, characterized in that: The device comprises: A clothing segmentation module is used to segment the input clothing image into various clothing regions according to a pre-trained clothing region segmentation model; A human body segmentation module is used to segment the input human image into various human body regions according to a pre-trained model region segmentation model; A region calculation module, used to calculate the union set of each human body region and the corresponding clothing region, and obtain each region to be deformed; The mapping module is used to map the clothing areas in each of the areas to be deformed to the corresponding human body areas to generate a virtual character map.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for generating a virtual character image as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for generating a virtual character image according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of any method described in claims 1 to 6 are implemented.