Method and system for enhancing sense of reality of virtual model on basis of generative network

Abstract

Provided in the present disclosure are a method and system for enhancing the sense of reality of a virtual model on the basis of a generative network. The method comprises: acquiring a rendered image of a virtual model wearing digital clothing; determining whether parsing information about the digital clothing in the rendered image is known; if the parsing information is unknown, performing extraction on the basis of a preset algorithm; converting the parsing information and then inputting converted parsing information into a generative network; under the control of a generative control condition, making the generative network generate a model character with a stronger sense of reality in a model area on the basis of model information; and converting the model character to obtain a generated image of a model wearing the digital clothing. By means of the solution in the present disclosure, on the basis of generative AI technology with high controllability, the sense of reality of clothing models can be significantly improved by fully utilizing priori information of digital clothing and priori information of a virtual model; and the high fidelity of modeled clothing is also kept, such that the solution is applied at a high adaptability to the alignment from the virtual model in a digital world to a real person model in a real world.

Description

A method and system for enhancing the realism of virtual models based on generative networks Technical Field

[0001] The present disclosure relates to the field of model enhancement, and in particular to a method and system for enhancing the realism of a virtual model based on a generative network. Background Art

[0002] A virtual mannequin is a specific application of 3D models. It refers to a computer-generated model that can be dressed up and performed in a digital environment. This technology can be used for clothing design and merchandising, as well as entertainment and artistic performances. Virtual mannequins are typically created using 3D modeling and animation techniques and can be produced and edited using a variety of software tools. Compared to real-life models, virtual mannequins are not restricted by time and location, allowing for costume changes and performances at any time. They also allow for more precise data analysis and testing.

[0003] Digital clothing is created, designed, and displayed using digital technology. It can be used for virtual fittings, online shopping, fashion design, gaming, and animation. Designed, simulated, and displayed using computer technology, digital clothing avoids the physical properties and manufacturing process considerations of traditional clothing production. It also facilitates modification and customization, offering users a more convenient, personalized, and cost-effective clothing purchasing and customization experience. It also provides fashion designers with a more flexible and efficient way to create and display digital clothing. The creation of digital clothing typically requires the use of tools such as 3D modeling software, graphics processing software, and virtual reality technology.

[0004] With the rapid development of the metaverse and virtual fashion, digital clothing and virtual models have become a key trend in the fashion industry. Digital modeling and design technologies enable designers to transform their creative ideas into digital models, allowing for rapid visualization and presentation of the resulting designs on virtual models. An increasing number of brands and designers are adopting digital technologies for clothing design and manufacturing. Digital clothing also presents new business opportunities and models, such as personalized customization and digital marketing.

[0005] While realistic digital clothing and model modeling, based on rendering technology, can realistically simulate the real world, there's still a significant gap compared to real models. Virtual models, in particular, often lack realistic and lifelike appearance and movement. Therefore, ensuring that digital clothing closely resembles modeled clothing while also enhancing the realism of virtual models, making them more like real models, has become a pressing technical challenge for the fashion industry.

[0006] Summary of the Invention

[0007] In view of this, the present disclosure proposes a method and system for enhancing the realism of virtual models based on a generative network. The specific scheme is as follows:

[0008] In the first part, this disclosure proposes a method for enhancing the realism of a virtual model based on a generative network, comprising the following steps:

[0009] Obtain renderings of virtual models wearing digital clothing;

[0010] Determining whether parsed information about the digital garment in the rendering is known, the parsed information including a garment region related only to the digital garment and a model region related only to the human body structure;

[0011] If unknown, the clothing area and model area are extracted based on the preset algorithm;

[0012] Converting the parsed information from the image pixel space to the latent space and then inputting it into the generative network;

[0013] Under the control of generation control conditions including model information, the generation network generates a more realistic model image in the model area based on the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing;

[0014] After converting the model image from the latent space to the image pixel space, a generated image of the model wearing the digital clothing is obtained.

[0015] In a specific embodiment, the generation control conditions also include clothing information, which includes the edges, contours, normals and colors of the digital clothing; through the clothing information, the generation network strengthens its understanding and cognition of the local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

[0016] In a specific embodiment, the model information includes posture, age, gender, skin color, face shape, facial features and hairstyle.

[0017] In a specific embodiment, the generative network includes a generative adversarial network and a diffusion model.

[0018] In a specific embodiment, encoding is performed by an image encoder to convert the parsed information from an image pixel space to a latent space; and decoding is performed by an image decoder matched with the image encoder to convert the model image from the latent space to an image pixel space.

[0019] In a specific embodiment, the clothing area is segmented based on image segmentation technology, the virtual model area is extracted based on human body analysis technology, and the model area is processed with different granularities adapted to different scenes based on image cutout technology.

[0020] In the second part, this disclosure proposes a virtual model reality enhancement system based on a generative network, which includes the following modules:

[0021] An input unit for acquiring a rendering of a virtual model wearing digital clothing;

[0022] a parsing information acquisition unit, configured to determine whether parsing information about the digital garment in the rendering image is known, the parsing information including a garment region related only to the digital garment and a model region related only to the human body structure; if not known, extracting the garment region and the model region based on a preset algorithm;

[0023] A first conversion unit, configured to convert the parsed information from the image pixel space to a latent space and then input the latent space into a generation network;

[0024] a generating unit, configured to, under the control of generating control conditions including model information, enable the generating network to generate a more realistic model image in the model area based on the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing;

[0025] The second conversion unit is used to convert the model image from the latent space to the image pixel space to obtain a generated image of the model wearing the digital clothing.

[0026] In a specific embodiment, the generation control condition further includes clothing information, and the clothing information includes edges, contours, normals and colors of the digital clothing;

[0027] The clothing information enables the generation network to strengthen its understanding and cognition of local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

[0028] In one embodiment, the model information includes posture, age, gender, skin color, face shape, facial features, and hairstyle; and / or, the generative network includes a generative adversarial network and a diffusion model;

[0029] And / or, the clothing area is segmented based on image segmentation technology, the virtual model area is extracted based on human body analysis technology, and the model area is processed with different granularities adapted to different scenes based on image cutout technology.

[0030] In a specific embodiment, encoding is performed by an image encoder to convert the parsed information from an image pixel space to a latent space; and decoding is performed by an image decoder matched with the image encoder to convert the model image from the latent space to an image pixel space.

[0031] In the third part, the present disclosure proposes a method for enhancing the realism of a virtual model based on a generative network, comprising the following steps:

[0032] receiving a rendering of a virtual model wearing the digital garment;

[0033] Extracting parsed information of the rendering, the parsed information including a clothing area and a model area;

[0034] The analysis information and the model information are input into a generation network to generate a model image wearing the digital clothing and corresponding to the model information.

[0035] In some specific embodiments, the clothing area is the area where the digital clothing is located, and the model area is the human body structure of the virtual model exposed outside the digital clothing. The model area does not include the human body structure covered by the digital clothing.

[0036] In some specific embodiments, before obtaining the parsed information of the rendering image, the method further includes:

[0037] Determining whether the rendering contains parsing information;

[0038] If the parsing information is not included, the clothing area and the model area are extracted based on a preset algorithm, and the edges of the clothing area and the model area are accurately expressed.

[0039] In some specific embodiments, the clothing area is extracted based on an image segmentation algorithm, the model area is extracted based on a human body analysis algorithm, and the edges of the clothing area and the model area are processed based on an image cutout algorithm to achieve accurate expression of the edges of the clothing area and the model area.

[0040] In some specific embodiments, the method of obtaining the model information includes:

[0041] Generate a reference model and use information parameters of the model as model information; and / or,

[0042] Through the configurable model custom input, the model configuration item text is converted into feature representation through the text encoder, and the feature representation of the model is used as the model information.

[0043] In some specific embodiments, the model region includes region contour information and human body part information.

[0044] In some specific embodiments, when the parsed information and preset model information are input into the generation network, it also includes: inputting clothing information into the generation network, wherein the clothing information includes one or more features of the clothing's edge, contour, normal, color, and texture.

[0045] In the fourth part, the present disclosure proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor;

[0046] The processor implements the method as described in any one of the third parts by running the executable instructions.

[0047] In the fifth part, the present disclosure proposes a computer-readable storage medium having computer instructions stored thereon, which implement the steps of any method described in the third part when executed by a processor.

[0048] Beneficial effects: The present disclosure provides a method and system for enhancing the realism of virtual models based on a generative network. Based on highly controllable generative AI technology, it makes full use of the prior information of digital clothing and virtual models, can significantly improve the realism of clothing models, while maintaining the high fidelity of modeled clothing, and is highly adaptable to the alignment of virtual models in the digital world to real-life models in the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] FIG1 is a flow chart of a method for enhancing the sense of reality according to an embodiment of the present disclosure;

[0050] FIG2 is a schematic diagram showing the principle of a method for enhancing the sense of reality according to an embodiment of the present disclosure;

[0051] FIG3 is a flow chart of another method for enhancing the sense of reality according to an embodiment of the present disclosure;

[0052] FIG4 is a schematic diagram of a module of an augmented reality system according to an embodiment of the present disclosure;

[0053] FIG5 is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0054] Figure numerals: 1-input unit; 2-parsed information acquisition unit; 3-first conversion unit; 4-generation unit; 5-second conversion unit; 702-processor; 704-internal bus; 706-network interface; 708-memory; 710-non-volatile memory. DETAILED DESCRIPTION

[0055] Hereinafter, various embodiments of the present disclosure will be described more fully. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but rather that the present disclosure should be understood to encompass all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of the present disclosure.

[0056] This embodiment discloses a method for enhancing the realism of virtual models based on a generative network, which can significantly improve the realism of clothing models while maintaining high fidelity of the modeled clothing. The specific process of the realism enhancement method is shown in Figure 1 of the specification, and the complete process is shown in Figure 2.

[0057] The specific plan is as follows:

[0058] A method for enhancing the realism of a virtual model based on a generative network comprises the following steps:

[0059] 101. Obtaining a rendering of a virtual model wearing digital clothing;

[0060] 102. Determine whether analytical information about the digital clothing in the rendering is known, the analytical information including the clothing region related only to the digital clothing and the model region related only to the human body structure; if unknown, extract the clothing region and the model region based on a preset algorithm;

[0061] 103. Convert the parsed information from the image pixel space to the latent space and input it into the generative network;

[0062] 104. Under the control of generation control conditions including model information, the generation network generates a more realistic model image in the model area based on the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing;

[0063] 105. After converting the model image from latent space to image pixel space, a generated image of the model wearing digital clothing is obtained.

[0064] The solution in this embodiment enhances the realism of the model in the rendering, while retaining the original fidelity of the clothing. A rendering is an image of a virtual scene or object created using computer graphics technology. Renderings can be used to create different visual effects by adjusting parameters such as the model, material, and lighting. The rendering in this embodiment is an image of a virtual model wearing digital clothing.

[0065] The solution of this embodiment requires detailed analysis of the rendered image to obtain analytical information about the virtual model wearing clothing. This analytical information includes the clothing region and the model region. The clothing region is the area where the digital clothing is located. The model region is the part of the virtual model's anatomy exposed outside the digital clothing. For example, if the rendered image shows a model wearing a dress, the entire dress area is the clothing region, and the exposed parts of the model's anatomy, such as the head, feet, and arms, are the model region. It is important to note that the model region does not represent the entire model. From another perspective, the model region and the clothing region together form a complete, clothed model. The separation of the clothing region and the model region is intended to maintain high fidelity of the digital clothing while ensuring that only the non-clothing areas of the model are enhanced for realism, while the clothing areas retain their original form. This embodiment separates the model region and the clothing region and feeds them into the generative network separately. This allows the generative network to process the model region and clothing region differently, making the overall system's image processing more flexible. Especially when computing power is limited, the system can more easily allocate computing resources to the areas that the user needs to enhance. Furthermore, the generative network can adopt different enhancement strategies for different areas, resulting in better enhancement results. In addition, further requirements are made for the separation of the model area and the clothing area. In this embodiment, the model area only needs to obtain the structures of various human body parts exposed outside the clothing, so that the generation network can selectively process the model and clothing separately. Specifically, this embodiment can support users to choose to enhance the realism of the model area or the clothing area, or enhance both. In the default preset environment, users can also choose the realism enhancement object according to the computing resources that the current system can mobilize. If the computing resources are not enough to enhance the realism of the model area and the clothing area at the same time, the system will give priority to enhancing the realism of the model area, and then the user can choose whether to continue to enhance the realism of the clothing. In some cases, this solution can also provide more detailed configuration options, supporting users to input different detailed information for the model area and the clothing area, and control the generation network to enhance the model area and the clothing area respectively. For example, the user can input more requirements for the realism enhancement of the model area, such as tattoos, skin color, scars and other details, and only input texture details for the clothing area. In this way, this solution only realizes the generation of details such as tattoos, skin color, scars and other details for the model area, and the clothing area only processes texture enhancement. Moreover, under the system of this embodiment, details belonging to the model area will not be generated in the clothing area in a disordered manner.

[0066] This embodiment proposes an adaptive, granular, garment-to-model separation technology solution for different virtual model sources, generating high-quality garment-to-model separation information. This makes the adaptability of digital garments and virtual models more complete and reliable, while also providing data support for enhancing the realism of virtual models.

[0067] 1. If the virtual model has its own analytical information of the digital clothing it is wearing, then subsequent processing will be carried out directly;

[0068] 2. When the rendered image's analytical information is unknown and only the avatar is available, extraction can only be performed through algorithms. Depending on the granularity, the following schemes are used, as shown in Figure 2.

[0069] Based on image segmentation technology, the clothing area is segmented, and the non-clothing model area is processed in the next step. Image segmentation refers to dividing an image into several regions with similar properties to facilitate better analysis and processing. Image segmentation technology can be applied to fields such as image recognition and target detection. The image segmentation techniques of this embodiment include threshold-based segmentation, region-based segmentation, and edge-based segmentation.

[0070] The threshold-based segmentation, region-based segmentation, and edge-based segmentation mentioned in this embodiment are only used to describe the various technical means that can be used in this solution to achieve the purpose of image segmentation. In addition, there are image segmentation based on deep learning, highlighting segmentation based on GAN, image segmentation based on diffusion model, etc. The main function of image segmentation is to isolate the clothing area, model area, and other non-clothing and non-model areas in the image, so as to facilitate the processing of the model area in the image, realize the preservation of clothing area information, and make the model area more consistent with the characteristics of the real model.

[0071] Based on human parsing technology, the virtual model area is extracted and processed in the next step. Human parsing is a more fine-grained image segmentation technology. In addition to decomposing the image into clothing area, model area, and other areas, it also further segments the model area with higher precision, such as hair, face, neck, arms, hands, feet, etc., to achieve fine-grained model understanding and analysis. Human parsing generally adopts traditional machine learning algorithms and deep learning algorithms, such as instance segmentation algorithms. For example, typically, the Mask RCNN algorithm is used to perform pixel-level fine-grained separation of each pixel in the image to extract the fine-grained semantic information of the model area. At the same time, a more accurate labeled data set is more important for the result. However, in general, it is understandable that there are many types of human parsing algorithms. This embodiment only lists some common algorithms. Those skilled in the art can choose the appropriate human parsing algorithm according to the desired effect.

[0072] Based on image matting technology, the virtual model area is finely extracted for the next step of processing. Image matting refers to the use of computer vision technology to identify and extract important features of the image, such as color, edges, texture, etc., to separate the part to be cut out from the background. In addition to understanding image information at the pixel level, the image matting algorithm also further accurately expresses the edges of clothing areas, model areas, and other areas. Compared with the segmentation algorithm, the edges of the area obtained by matting are softer and can express the regional effect with different levels of transparency. This is especially important for hair areas.

[0073] In addition, algorithms based on large models, such as SAM (Segment Anything), draw on the prompt idea in NLP tasks. By learning from large amounts of data, they can achieve rapid segmentation of target objects through prompt input, but it is still an image segmentation technology.

[0074] This embodiment selects three different algorithms to distinguish between clothing and model regions, as well as the division of human anatomy. This results in better generation results and, since it doesn't process global details, consumes minimal computing resources. This embodiment allows for targeted realism enhancement of individual model parts, necessitating the introduction of a human body parsing algorithm to accurately identify individual body parts. This is necessary to support users' needs for customized model enhancement. For example, in real life, a striking model often possesses unique features on their body or face that complement the clothing. A readily understood example is the interplay between personalized tattoos and trendy streetwear, which enhances the overall visual effect. Therefore, in digital scenarios, this solution also needs to support users in customizing the design of these model and clothing elements. For example, if a user desires a tattoo on the model's left arm, a human body parsing algorithm can help the system identify the left arm, thereby controlling the generation network to generate the tattoo only on the model's left arm, rather than the right. This embodiment accurately breaks down the image and enhances the details to process and output high-fidelity fitting renderings. Because high fidelity is required, the system needs to invest computing resources in areas that require greater realism. In addition, the image cutout algorithm ensures that the image maintains its integrity, giving the entire image a better visual effect.

[0075] After obtaining the clothing area and the model area, they can be sent to the generative network for realistic generation of the virtual model. In this embodiment, before the parsed information is input into the generative network, the network calculation needs to be converted from the image pixel space to the latent space to achieve data dimensionality reduction and compression, and reduce the amount of calculation. The main method is, for example, using a VAE encoder for processing. After the generative network generates the model image, it also needs to be converted from the latent space to the image pixel space. The two conversion steps can significantly reduce the amount of calculation. In a specific embodiment, encoding is performed by an image encoder to convert the parsed information from the image pixel space to the latent space; decoding is performed by an image decoder matched with the image encoder to convert the model image from the latent space to the image pixel space. Specific details are shown in Figure 2.

[0076] Preferably, the present embodiment mainly adopts a generation method based on a diffusion model. The diffusion model is a relatively advanced machine learning algorithm that gradually adds random noise to the data and then learns the reverse diffusion process to generate high-quality data samples from the noise. During the generation process, an image with coherent structure and rich details is generated through repeated diffusion and denoising processes in an iterative manner. The diffusion model is a generation model based on diffusion, which generates images through multiple stages. At each stage, the model adds some noise to the input noise, thereby gradually changing the content of the image. Through multiple iterations, a high-quality image can eventually be generated. In order to better characterize high-quality model generation, a large-scale training data set is constructed to be able to more effectively output realistic model effects.

[0077] Generative Adversarial Networks (GANs) can also achieve the goal of generative tasks. GANs are unsupervised learning models that generate realistic images through adversarial training between a generator network and a discriminator network. The generator network attempts to generate images similar to the training data, while the discriminator network attempts to distinguish between the generator's images and the real images in the training data. Through this adversarial training, GANs can generate high-quality images. The diffusion model is a diffusion-based generative model that generates images through multiple stages. In each stage, the model adds noise to the input, gradually changing the image's content. Through multiple iterations, high-quality images are ultimately generated.

[0078] The generative network requires additional generative control conditions to control the generation of the model image and ensure the controllability of the generation process. These control conditions involve both model and clothing information. By using clothing information to maintain the overall effect of the clothing, the generative network strengthens its understanding and cognition of the local details and global information of the digital clothing and model, effectively processing the digital model in a targeted manner while maintaining high fidelity of the design details and elements of the digital clothing. Local details and global information are extracted through local region feature extraction in the image, such as the clothing area and the model area, to understand local characteristics. Global information is a global understanding of the entire image, such as the fit between the model and clothing and the naturalness of the presentation.

[0079] Model information is used to ensure that the newly generated model matches the model in the input image in pose and position. Clothing information specifically includes details about clothing edges, contours, normals, colors, and textures, while model information encompasses various aspects such as pose, age, gender, skin color, face shape, facial features, and hairstyle. Any features that could influence the model's appearance can be incorporated into the model information, effectively enhancing the realism of the virtual model's image. Furthermore, this embodiment supports more pre-analysis and processing information as generation control conditions, injecting more prior knowledge into generative AI to further enhance its controllability and authenticity. Here, to achieve model controllability, two main approaches are used to edit features such as posture, age, gender, skin color, face shape, facial features, and hairstyle. One approach is to leverage the capabilities of digital modeling software to predefine a digital model that meets the requirements during model generation. The other approach is to use configurable model customization input, such as a text input field, to convert the model configuration text into feature representations based on a text encoder. Furthermore, through feature fusion methods such as feature splicing and cross-attention, the model generation effect can be controlled and adjusted. These two approaches can be used either or simultaneously. Similarly, clothing information can also be generated by generating a reference clothing model, using the information parameters of this model as clothing information, and / or by using configurable clothing customization input, converting the clothing configuration text into feature representations based on a text encoder. However, clothing information is not required. This depends on whether the user needs to further enhance the realism of the clothing through clothing information. For example, in cases where computing power is insufficient, the user can choose not to input clothing information and rely solely on the model information to enhance the realism of the model. Alternatively, if the user feels that the clothing also needs to be more realistic, clothing information can be input into the generation network as well. This makes the system more interactive and allows users to freely choose how to input control conditions. If users have their own model or want to edit one first, the system provides the function of using the model's parameters as control conditions, so that users have greater control over the details of the generated network. If users want to customize the input through the model, there is no need to edit the model in advance, which is easier for users to operate and can achieve the desired effect more quickly.

[0080] At the same time, if the user has very high requirements for the realism and fineness of some human body details, but not for other human body details, the above two methods can be used at the same time to control the generative network. For example, the user has a special personalized design for the tattoo on the model's left arm, and hopes that the image output by the generative network will be retained as it is, while there are no special requirements for the tattoo on the model's right arm, and only a tattoo is needed to provide a visual effect. In this way, the user can input the model's left arm tattoo into the generative network through model parameters, and input the model's right arm tattoo in the form of text. In this way, on the model rendering processed by the generative network, the tattoo on the model's left arm will be retained as it is, while a tattoo roughly in line with the user's expectations will be generated on the right arm; the same is true for enhancing the realism of clothing.

[0081] This embodiment also discloses another method for enhancing the realism of a virtual model based on a generative network. The process is shown in FIG3 and specifically includes the following steps:

[0082] 301. Receive a rendering of a virtual model wearing digital clothing;

[0083] 302. Extracting parsed information of the rendering, where the parsed information includes a clothing area and a model area;

[0084] 303. Input the parsed information and the model information into the generation network to generate a model image wearing the digital clothing and corresponding to the model information.

[0085] In this embodiment, the model area and the clothing area are first distinguished and then input into the generative network. In this way, the generative network can have different processing methods for these two different areas, making the way the entire system processes images more flexible. Especially when computing power is limited, the system can more easily invest computing resources in specific areas. At the same time, the generative network can adopt different enhancement strategies for different areas, which will achieve better enhancement effects.

[0086] In some specific embodiments, the clothing region is the area where the digital clothing is located, and the model region is the human body structure of the virtual model exposed outside the digital clothing. The model region does not include human body structure covered by the digital clothing. In some specific embodiments, the model region includes regional contour information and human body part information. Further requirements are made for the separation of the model region and clothing region. In this embodiment, the model region only needs to obtain the various human body parts exposed outside the clothing, so that the generation network can selectively process the model region and clothing region separately. Specifically, this embodiment can support users to choose to enhance the realism of the model area or the clothing area, or enhance both. In addition, under the default preset environment, the system can also select the realism enhancement object according to the computing resources that the current system can mobilize. For example, in this embodiment, if the computing resources are not enough to enhance the realism of the model area and the clothing area at the same time, the system will give priority to enhancing the realism of the model area, and then the user can choose whether to continue to enhance the realism of the clothing. In some cases, the system can also provide more detailed configuration options, supporting users to input different detailed information for the model area and the clothing area, and control the generation network to enhance the model area and the clothing area respectively. For example, the user can input more requirements for the realism enhancement of the model area, such as tattoos, skin color, scars and other details, and only input texture details for the clothing area. In this way, the system only generates details such as tattoos, skin color, scars and other details for the model area, and the clothing area only processes texture enhancement. Moreover, under the system of this embodiment, details belonging to the model area will not be generated in the clothing area in a disordered manner.

[0087] In some specific embodiments, before obtaining the analytical information of the rendering, it also includes: determining whether the rendering contains analytical information; if it does not contain analytical information, extracting the clothing area and the model area based on a preset algorithm; and accurately expressing the edges of the clothing area and the model area.

[0088] In some specific embodiments, clothing regions are extracted using an image segmentation algorithm, model regions are extracted using a human body analysis algorithm, and edges of the clothing and model regions are processed using an image cutout algorithm to achieve precise representation of these regions. Three different approaches were selected to address the details involved in clothing, model, and human anatomy, resulting in better representation of these details. This approach aims to accurately decompose the image and enhance details to produce high-fidelity fitting renderings. However, inaccurate human anatomy recognition cannot meet market demands. From another perspective, precisely because of the need for high fidelity, the system must invest limited computing resources in more critical areas. This problem is not present in rendering-related fields, such as the gaming industry. Furthermore, the image cutout algorithm ensures the integrity of the image, enhancing the overall visual quality. This step is not necessary for optimizing only the face, as it does not require consideration of the impact of clothing on the face or the enhancement of the texture and individuality of other parts.

[0089] In some specific embodiments, obtaining model information includes: generating a reference model model and using the model's parameters as model information; and / or, using a configurable model customization input, converting model configuration text into a feature representation using a text encoder, and using the model's feature representation as model information. Model information, as a core element of model generation, supports user model customization or semantic input. Clothing information, as a reference for providing global information, focuses on only edge information.

[0090] In some specific embodiments, when the parsing information and the preset model information are input into the generation network, it also includes: inputting the clothing information into the generation network, where the clothing information includes one or more features of the clothing's edge, contour, normal, color, and texture.

[0091] This embodiment provides a method for enhancing the realism of virtual models based on a generative network. Based on highly controllable generative AI technology, it fully utilizes the prior information of digital clothing and virtual models, can significantly improve the realism of clothing models, while maintaining the high fidelity of modeled clothing, and is highly adaptable to aligning virtual models in the digital world with real-world models.

[0092] The disclosed embodiments also disclose a generative network-based virtual model reality enhancement system, which systematizes the generative network-based virtual model reality enhancement method described in the above embodiment to make it more practical. The overall structure of the generative network-based virtual model reality enhancement system is shown in Figure 4 of the specification. The specific scheme is as follows:

[0093] A virtual model reality enhancement system based on a generative network includes the following modules:

[0094] An input unit 1 is configured to obtain a rendering of a virtual model wearing digital clothing;

[0095] Parsed information acquisition unit 2, used to determine whether parsed information about the digital clothing in the rendering image is known, the parsed information including only the clothing area related to the digital clothing and only the model area related to the human body structure;

[0096] If unknown, the clothing area and model area are extracted based on the preset algorithm;

[0097] A first conversion unit 3 is used to convert the parsed information from the image pixel space to the latent space and then input it into the generation network;

[0098] A generation unit 4 is configured to enable the generation network to generate a more realistic model image in the model area based on the model information under the control of generation control conditions including the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing;

[0099] The second conversion unit 5 is used to convert the model image from the latent space to the image pixel space to obtain a generated image of the model wearing the digital clothing.

[0100] Among them, the generation control conditions also include clothing information related to the edges, contours, normals and colors of the digital clothing; through clothing information, the generation network strengthens its understanding and cognition of the local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

[0101] Preferably, the model information includes age, gender, skin color, face shape, facial features and hairstyle; and or, the generative network includes a generative adversarial network and a diffusion model; and / or, the clothing area is segmented based on image segmentation technology, the virtual model area is extracted based on human body analysis technology, and the model area is processed with different granularities to adapt to different scenes based on image cutout technology.

[0102] Preferably, encoding is performed by an image encoder to convert the parsing information from the image pixel space to the latent space; and decoding is performed by an image decoder matched with the image encoder to convert the model image from the latent space to the image pixel space.

[0103] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of a preferred implementation scenario, and that the modules or processes depicted in the drawings are not necessarily required for implementing the present disclosure. Those skilled in the art will appreciate that the modules within the devices within the implementation scenarios may be distributed throughout the devices within the implementation scenarios as described in the implementation scenarios, or may be modified accordingly and located within one or more devices distinct from the implementation scenarios.

[0104] An embodiment of this specification also provides an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor implements any of the above method embodiments by running the executable instructions.

[0105] FIG5 is a schematic diagram of the structure of a device provided by an exemplary embodiment. Referring to FIG5 , at the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0106] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0107] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0108] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0109] The embodiments of this specification also provide a computer-readable storage medium having computer instructions stored thereon, which implement the steps of any of the above method embodiments when executed by a processor.

[0110] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0111] The present disclosure provides a method and system for enhancing the realism of virtual models based on a generative network. Based on highly controllable generative AI technology, it fully utilizes the prior information of digital clothing and virtual models, can significantly improve the realism of clothing models, while maintaining the high fidelity of modeled clothing, and is highly adaptable to aligning virtual models in the digital world with real-world models.

[0112] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0113] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0114] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0115] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0116] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A method for enhancing the sense of reality of a virtual model based on a generative network, characterized in that: The steps include: Get renderings of virtual models wearing digital clothing; Determining whether analytical information about the digital clothing in the rendering is known, the analytical information including a clothing area related only to the digital clothing and a model area related only to the human body structure; If unknown, the clothing area and model area are extracted based on a preset algorithm; The parsed information is converted from the image pixel space to the latent space and then input into the generation network; Under the control of generation control conditions including model information, the generation network generates a model image with a stronger sense of reality in the model area based on the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing; After converting the model image from latent space to image pixel space, a generated image of the model wearing the digital clothing is obtained.

2. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: The generation control conditions also include clothing information, which includes the edge, outline, normal and color of the digital clothing; The clothing information enables the generation network to strengthen its understanding and cognition of local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

3. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: The model information includes posture, age, gender, skin color, face shape, facial features and hairstyle.

4. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: The generative network includes a generative adversarial network and a diffusion model.

5. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: Encoding by an image encoder to convert the parsed information from an image pixel space to a latent space; The model image is decoded by an image decoder matched with the image encoder to transform the model image from the latent space into the image pixel space.

6. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: The clothing area is segmented based on the image segmentation technology, the virtual model area is extracted based on the human body analysis technology, and the model area is processed with different granularities to adapt to different scenes based on the image cutout technology.

7. A virtual model reality enhancement system based on a generative network, characterized in that: Includes the following modules: An input unit, for obtaining a rendering of a virtual model wearing digital clothing; a parsing information acquisition unit, configured to determine whether parsing information about the digital clothing in the rendering is known, the parsing information including a clothing area related only to the digital clothing and a model area related only to the human body structure; If unknown, the clothing area and model area are extracted based on a preset algorithm; A first conversion unit, used to convert the parsing information from the image pixel space to the latent space and then input it into the generation network; A generating unit, configured to enable the generating network to generate a model image with a stronger sense of reality in the model area based on the model information under the control of generating control conditions including the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing; The second conversion unit is used to convert the model image from the latent space to the image pixel space to obtain a generated image of the model wearing the digital clothing.

8. The virtual model reality enhancement system according to claim 7, characterized in that: The generation control conditions also include clothing information, which includes the edge, outline, normal and color of the digital clothing; The clothing information enables the generation network to strengthen its understanding and cognition of local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

9. The virtual model reality enhancement system according to claim 7, characterized in that: The model information includes posture, age, gender, skin color, face shape, facial features and hairstyle; And / or, the generative network includes a generative adversarial network and a diffusion model; And / or, the clothing area is segmented based on the image segmentation technology, the virtual model area is extracted based on the human body analysis technology, and the model area is processed with different granularities adapted to different scenes based on the image cutout technology.

10. The virtual model reality enhancement system according to claim 7, characterized in that: Encoding by an image encoder to convert the parsed information from an image pixel space to a latent space; The model image is decoded by an image decoder matched with the image encoder to transform the model image from the latent space into the image pixel space.

11. A method for enhancing the realism of a virtual model based on a generative network, characterized in that: The steps include: receiving a rendering of a virtual model wearing the digital garment; Extracting parsed information of the rendering, the parsed information including a clothing area and a model area; The analysis information and the model information are input into a generation network to generate a model image wearing the digital clothing and corresponding to the model information.

12. The method for enhancing the sense of reality of a virtual model according to claim 11, characterized in that: The clothing area is the area where the digital clothing is located, and the model area is the human body structure of the virtual model exposed outside the digital clothing. The model area does not include the human body structure covered by the digital clothing.

13. The method for enhancing the sense of reality of a virtual model according to claim 11, characterized in that: Before obtaining the parsing information of the rendering, it also includes: Determining whether the rendering contains parsing information; If the analysis information is not included, the clothing area and the model area are extracted based on a preset algorithm, and the edges of the clothing area and the model area are accurately expressed.

14. The method for enhancing the sense of reality of a virtual model according to claim 13, characterized in that: The clothing area is extracted based on an image segmentation algorithm, the model area is extracted based on a human body analysis algorithm, and the edges of the clothing area and the model area are processed based on an image cutout algorithm to achieve accurate expression of the edges of the clothing area and the model area.

15. The method for enhancing the reality of a virtual model according to claim 11, characterized in that: Methods for obtaining the model information include: Generate a reference model and use information parameters of the model as model information; and / or, Through the configurable model custom input, the model configuration item text is converted into feature representation through a text encoder, and the feature representation of the model is used as the model information.

16. The method for enhancing the sense of reality of a virtual model according to claim 11, characterized in that: The model region includes region contour information and human body part information.

17. The method for enhancing the reality of a virtual model according to claim 11, characterized in that: When the analysis information and the preset model information are input into the generation network, the method further includes: inputting clothing information into the generation network, wherein the clothing information includes one or more features of the edge, contour, normal, color, and texture of the clothing.

18. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 11 to 17 by running the executable instructions.

19. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 11 to 17 are implemented.

Images