Virtual model reality enhancement method and system based on generative network

Abstract

The invention provides a virtual model reality enhancement method and system based on a generative network. The method comprises the steps that a rendering graph of a virtual model wearing digital clothes is acquired; judging whether analysis information about the digital clothing in the rendered image is known or not; if not, extracting based on a preset algorithm; converting the analysis information, and inputting the converted analysis information into a generative network; under the control of the generation control condition, enabling the generation network to generate a model image with a stronger sense of reality in the model area based on the model information; and converting the model image to obtain a model generation graph of the digital clothing. The scheme disclosed by the invention is based on a high-controllability generative AI technology, priori information of the digital clothing and the virtual model is fully utilized, the sense of reality of the clothing model can be remarkably improved, meanwhile, high fidelity of the modeled clothing is kept, and the method is applied to alignment from the digital world virtual model to the real world real human model in a high-adaptability manner.

Description

Technical Field

[0001] The present disclosure relates to the field of model enhancement, and in particular to a method and system for enhancing the sense of reality of a virtual model based on a generative network. Background Art

[0002] A virtual model is a specific application of a 3D model. It refers to a model generated using computer technology that can change clothes and perform in a digital environment. This technology can be used for clothing design and sales, as well as entertainment and artistic performances. Virtual models are usually created through 3D modeling and animation technology, and can be produced and edited using a variety of software tools. Compared with real models, virtual models are not restricted by time and place, and can be changed and performed at any time. At the same time, more accurate data analysis and testing can also be performed.

[0003] Digital clothing is a type of clothing created, designed and displayed using digital technology. It can be used for virtual fitting, online shopping, fashion design, games and animation, etc. Digital clothing is designed, simulated and displayed through computer technology, which can avoid the physical properties and manufacturing processes that need to be considered in the traditional clothing production process. It is also convenient for modification and customization, providing users with a more convenient, personalized and low-cost clothing purchase and customization experience, and can also provide clothing designers with a more flexible and efficient way of creation and display. The creation of digital clothing usually requires the use of tools such as 3D modeling software, graphics processing software, and virtual reality technology.

[0004] With the rapid development of the metaverse and virtual fashion, digital clothing and virtual models have become an important trend in the fashion industry. The use of digital modeling and design technology can transform the creativity of fashion designers into digital models, and quickly view and display the effects on virtual models. More and more brands and designers are beginning to use digital technology for clothing design and manufacturing. At the same time, digital clothing has also brought new business opportunities and business models, such as personalized customization, digital marketing, etc.

[0005] Although realistic digital clothing and model modeling can simulate the real world more realistically based on rendering technology, there is still a big gap compared with real models, especially virtual models, which generally have the problem of not being realistic and vivid in appearance and movements. Therefore, how to improve the realism of virtual models while ensuring that digital clothing is as close to the modeling effect as possible, so that virtual models can be closer to real models, has become a technical problem that the fashion industry urgently needs to solve. Summary of the invention

[0006] In view of this, the present disclosure proposes a method and system for enhancing the sense of reality of a virtual model based on a generative network, and the specific scheme is as follows:

[0007] In the first part, the present disclosure proposes a method for enhancing the realism of a virtual model based on a generative network, comprising the following steps:

[0008] Get renderings of virtual models wearing digital clothing;

[0009] Determining whether analytical information about the digital clothing in the rendering is known, the analytical information including a clothing area related only to the digital clothing and a model area related only to the human body structure;

[0010] If unknown, the clothing area and model area are extracted based on a preset algorithm;

[0011] The parsed information is converted from the image pixel space to the latent space and then input into the generation network;

[0012] Under the control of generation control conditions including model information, the generation network generates a model image with a stronger sense of reality in the model area based on the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing;

[0013] After converting the model image from latent space to image pixel space, a generated image of the model wearing the digital clothing is obtained.

[0014] In a specific embodiment, the generation control condition further includes clothing information, and the clothing information includes the edge, contour, normal and color of the digital clothing;

[0015] The clothing information enables the generation network to strengthen its understanding and cognition of local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

[0016] In a specific embodiment, the model information includes posture, age, gender, skin color, face shape, facial features and hairstyle.

[0017] In a specific embodiment, the generative network includes a generative adversarial network and a diffusion model.

[0018] In a specific embodiment, encoding is performed by an image encoder to convert the parsed information from an image pixel space to a latent space;

[0019] The model image is decoded by an image decoder matched with the image encoder to transform the model image from the latent space into the image pixel space.

[0020] In a specific embodiment, the clothing area is segmented based on the image segmentation technology, the virtual model area is extracted based on the human body analysis technology, and the model area is processed with different granularities to adapt to different scenes based on the image cutout technology.

[0021] In the second part, the present disclosure proposes a virtual model reality enhancement system based on a generative network, including the following modules:

[0022] An input unit, for obtaining a rendering of a virtual model wearing digital clothing;

[0023] a parsing information acquisition unit, configured to determine whether parsing information about the digital clothing in the rendering is known, the parsing information including a clothing area related only to the digital clothing and a model area related only to the human body structure;

[0024] If unknown, the clothing area and model area are extracted based on a preset algorithm;

[0025] A first conversion unit, used to convert the parsing information from the image pixel space to the latent space and then input it into the generation network;

[0026] A generating unit, configured to enable the generating network to generate a model image with a stronger sense of reality in the model area based on the model information under the control of generating control conditions including the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing;

[0027] The second conversion unit is used to convert the model image from the latent space to the image pixel space to obtain a generated image of the model wearing the digital clothing.

[0028] In a specific embodiment, the generation control condition further includes clothing information, and the clothing information includes the edge, contour, normal and color of the digital clothing;

[0029] The clothing information enables the generation network to strengthen its understanding and cognition of local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

[0030] In a specific embodiment, the model information includes posture, age, gender, skin color, face shape, facial features and hairstyle;

[0031] And or, the generative network includes a generative adversarial network and a diffusion model;

[0032] And / or, the clothing area is segmented based on the image segmentation technology, the virtual model area is extracted based on the human body analysis technology, and the model area is processed with different granularities adapted to different scenes based on the image cutout technology.

[0033] In a specific embodiment, encoding is performed by an image encoder to convert the parsed information from an image pixel space to a latent space;

[0034] The model image is decoded by an image decoder matched with the image encoder to transform the model image from the latent space into the image pixel space.

[0035] In the third part, the present disclosure proposes a method for enhancing the realism of a virtual model based on a generative network, comprising the following steps:

[0036] receiving a rendering of a virtual model wearing the digital garment;

[0037] Extracting parsed information of the rendering, the parsed information including a clothing area and a model area;

[0038] The analysis information and the model information are input into a generation network to generate a model image wearing the digital clothing and corresponding to the model information.

[0039] In some specific embodiments, the clothing area is the area where the digital clothing is located, the model area is the human body structure of the virtual model exposed outside the digital clothing, and the model area does not include the human body structure covered by the digital clothing.

[0040] In some specific embodiments, before obtaining the parsed information of the rendering image, the method further includes:

[0041] Determining whether the rendering contains parsing information;

[0042] If the analysis information is not included, the clothing area and the model area are extracted based on a preset algorithm, and the edges of the clothing area and the model area are accurately expressed.

[0043] In some specific embodiments, the clothing area is extracted based on an image segmentation algorithm, the model area is extracted based on a human body analysis algorithm, and the edges of the clothing area and the model area are processed based on an image cutout algorithm to achieve accurate expression of the edges of the clothing area and the model area.

[0044] In some specific embodiments, the method of obtaining the model information includes:

[0045] Generate a reference model and use information parameters of the model as model information; and / or,

[0046] Through the configurable model custom input, the model configuration item text is converted into feature representation through a text encoder, and the feature representation of the model is used as the model information.

[0047] In some specific embodiments, the model region includes region contour information and human body part information.

[0048] In some specific embodiments, when the parsed information and preset model information are input into the generation network, it also includes: inputting clothing information into the generation network, wherein the clothing information includes one or more features of the clothing's edge, contour, normal, color, and texture.

[0049] In the fourth part, the present disclosure provides an electronic device, including:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor implements the method as described in any one of the third parts by running the executable instructions.

[0053] In the fifth part, the present disclosure proposes a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any method described in the third part.

[0054] Beneficial effects:

[0055] The present disclosure provides a method and system for enhancing the sense of reality of a virtual model based on a generative network. Based on a highly controllable generative AI technology, the method makes full use of the prior information of digital clothing and virtual models, can significantly improve the sense of reality of clothing models, while maintaining the high fidelity of modeled clothing, and is highly adaptable to the alignment of virtual models in the digital world to real-life models in the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a method for enhancing the sense of reality according to an embodiment of the present disclosure;

[0057] Figure 2 is a schematic diagram of the principle of the method for enhancing the sense of reality according to an embodiment of the present disclosure;

[0058] Figure 3 is another flow chart of a method for enhancing the sense of reality according to an embodiment of the present disclosure;

[0059] Figure 4 is a schematic diagram of a module of an enhanced reality system according to an embodiment of the present disclosure;

[0060] Figure 5 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0061] Figure numerals: 1-input unit; 2-parsed information acquisition unit; 3-first conversion unit; 4-generation unit; 5-second conversion unit; 702-processor; 704-internal bus; 706-network interface; 708-memory; 710-non-volatile memory. DETAILED DESCRIPTION

[0062] In the following, various embodiments of the present disclosure will be described more fully. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0063] This embodiment discloses a method for enhancing the sense of reality of a virtual model based on a generative network, which can significantly improve the sense of reality of a clothing model while maintaining the high fidelity of the modeled clothing. The specific process of the method for enhancing the sense of reality is shown in the attached manual. Figure 1 The complete process is shown in the attached Figure 2 shown.

[0064] The specific plan is as follows:

[0065] A method for enhancing the sense of reality of a virtual model based on a generative network comprises the following steps:

[0066] 101. Obtaining a rendering of a virtual model wearing digital clothing;

[0067] 102. Determine whether analytical information about the digital clothing in the rendering is known, the analytical information including a clothing area related only to the digital clothing and a model area related only to the human body structure; if unknown, extract the clothing area and the model area based on a preset algorithm;

[0068] 103. Convert the parsed information from the image pixel space to the latent space and input it into the generative network;

[0069] 104. Under the control of generation control conditions including model information, the generation network generates a model image with a stronger sense of reality in the model area based on the model information, wherein the model image maintains the form of the virtual model and wears digital clothing;

[0070] 105. After converting the model image from latent space to image pixel space, a generated image of the model wearing digital clothing is obtained.

[0071] The solution of this embodiment is to enhance the realism of the model in the rendering, while the clothing retains its original fidelity. A rendering is an image of a virtual scene or object created by computer graphics technology. A rendering can generate different visual effects by adjusting parameters such as models, materials, and lighting. The rendering of this embodiment is an image of a virtual model wearing digital clothing.

[0072] The solution of this embodiment requires a refined analysis of the rendering to obtain the analysis information of the virtual model wearing clothes. The analysis information includes clothing area and model area. The clothing area is the area where the digital clothing is located. The model area is the human body structure of the virtual model exposed outside the digital clothing. For example, in the rendering, the model is wearing a dress, then the area where the dress is located is the clothing area, and the organs of the model's body structure exposed to the outside, such as the head, feet, and arms, are the model area. It should be noted that the model area is not a complete model. From another perspective, the model area and the clothing area are combined to form a complete model wearing clothes. The separation of the clothing area and the model area is to maintain the high fidelity of the digital clothing, while ensuring that only the model in the non-clothing area is enhanced in realism, while the clothing area still maintains its original form. In this embodiment, the model area and the clothing area are first distinguished and input into the generation network separately, so that the generation network can have different processing methods for the model area and the clothing area, making the way the entire system processes images more flexible, especially in the case of tight computing power, the system is more likely to invest computing resources in the areas that users need to focus on, and at the same time, the generation network can adopt different enhancement strategies for different areas, which will make the enhancement effect better. In addition, further requirements are made for the separation of the model area and the clothing area. In this embodiment, the model area only needs to obtain the structure of each human body part exposed outside the clothing, so that the generation network can selectively process the model and clothing separately. Specifically, this embodiment can support users to choose to enhance the sense of reality of the model area or the clothing area, or enhance both. In the default preset environment, users can also select the object of sense of reality enhancement according to the computing resources that the current system can mobilize. If the computing resources are not enough to enhance the sense of reality of the model area and the clothing area at the same time, the system will give priority to enhancing the sense of reality of the model area, and then the user can choose whether to continue to enhance the sense of reality of the clothing. In some cases, this solution can also provide more detailed configuration options, support users to input different detail information for the model area and the clothing area, and control the generation network to enhance the model area and the clothing area respectively. For example, the user can input more requirements for the sense of reality enhancement of the model area, such as tattoos, skin color, scars and other details, and only input texture details for the clothing area. In this way, this solution only realizes the generation of details such as tattoos, skin color, scars and the like for the model area, and the clothing area only processes the enhancement of texture, and under the system of this embodiment, the details belonging to the model area will not be generated in the clothing area in a disordered manner.

[0073] This embodiment proposes adaptive and different-granularity clothing and model separation technology solutions for different virtual model sources to obtain high-quality clothing and model separation information, making the adaptability of digital clothing and virtual models more complete and reliable. At the same time, it provides data support for enhancing the realism of virtual models.

[0074] 1. If the virtual model has its own analytical information of wearing digital clothing, the subsequent processing will be carried out directly;

[0075] 2. When the analytical information in the rendering is unknown and only the virtual model is available, the extraction can only be done through algorithms. According to different granularities, the following solutions are adopted, as shown in the attached Figure 2 shown.

[0076] Based on the image segmentation technology, the clothing area is segmented, and the model part of the non-clothing area is processed in the next step. Image segmentation refers to dividing an image into several areas with similar properties for better analysis and processing. Image segmentation technology can be applied to image recognition, target detection and other fields. The image segmentation technology of this embodiment includes threshold-based segmentation, region-based segmentation, and edge-based segmentation.

[0077] The threshold-based segmentation, region-based segmentation, edge-based segmentation, etc. mentioned in this embodiment are only used to describe the various technical means that can be selected in this solution to achieve the purpose of image segmentation. In addition, there are image segmentation based on deep learning, highlighting segmentation based on GAN, image segmentation based on diffusion model, etc. The role of image segmentation is mainly to isolate the clothing area, model area, and other non-clothing and non-model areas in the image, so as to facilitate the processing of the model area in the image, realize the preservation of clothing area information, and the model area is more in line with the characteristics of the real model.

[0078] Based on human parsing technology, the virtual model area is extracted for further processing. Human parsing is a more fine-grained image segmentation technology. In addition to decomposing the image into clothing area, model area, and other areas, the model area will also be further segmented with higher precision, such as hair, face, neck, arms, hands, feet, etc., to achieve fine-grained model understanding and analysis. Human parsing generally adopts traditional machine learning algorithms and deep learning algorithms, such as instance segmentation algorithms. For example, typically, each pixel in the image is separated at pixel level based on the Mask RCNN algorithm to extract the fine-grained semantic information of the model area. At the same time, a more accurate annotated data set is more important for the result. But overall, it can be understood that there are many types of human parsing algorithms. This embodiment only lists some common algorithms. Those skilled in the art can choose the appropriate human parsing algorithm according to the desired effect.

[0079] Based on image matting technology, the virtual model area is refined and processed for the next step. Image matting refers to the use of computer vision technology to separate the part to be cut from the background by identifying and extracting important features of the image, such as color, edges, texture, etc. In addition to understanding image information at the pixel level, the image matting algorithm also further accurately expresses the edges of clothing areas, model areas, and other areas. Compared with the segmentation algorithm, the edges of the area obtained by matting are softer and can express the regional effect in different transparencies, which is especially important for hair areas.

[0080] In addition, algorithms based on large models, such as SAM (Segment Anything), draw on the prompt idea in NLP tasks and, by learning from large amounts of data, can achieve rapid segmentation of target objects through prompt input, but it is still an image segmentation technology.

[0081] This embodiment selects three different algorithms for processing for the distinction between clothing areas and model areas, as well as the division of human body structure, which will make the generation effect better, and because it does not process global details, the computing resources consumed are also very small. This embodiment can enhance the sense of reality of each part of the model in a targeted manner, so it is necessary to introduce a human body analysis algorithm to accurately identify each part of the human body, so as to support the user's demand for customized detail enhancement of the model. For example, in reality, an impressive model often has some personalized features on the body or face, and the existence of these features can just form an echo relationship with clothing. An easy-to-understand example is the mutual echo of personalized tattoos and trendy street clothing, which will make the overall visual effect better. Therefore, in the digital scene, this solution also needs to support users to customize the design of these models and clothing elements. For example, users may need to design a tattoo pattern on the left arm of the model. The human body analysis algorithm can help the system realize the recognition of the left arm of the human body, so that the generation network can be controlled to generate tattoos only on the left arm of the model, rather than on the right arm. This embodiment accurately disassembles the image and enhances the details to process and output high-fidelity fitting renderings. Because high fidelity is required, the system needs to invest computing resources in areas that need to improve realism. In addition, the image cutout algorithm ensures that the image always maintains its integrity, making the visual effect of the entire image better.

[0082] After obtaining the clothing area and the model area, they can be sent to the generation network for realistic generation of the virtual model. In this embodiment, before the parsed information is input into the generation network, it is necessary to convert the network calculation from the image pixel space to the latent space to achieve data dimension reduction and compression and reduce the amount of calculation. The main method is, for example, using a VAE encoder for processing. After the generation network generates the model image, it also needs to be converted from the latent space to the image pixel space. The two conversion steps can significantly reduce the amount of calculation. In a specific embodiment, encoding is performed through an image encoder to convert the parsed information from the image pixel space to the latent space; decoding is performed through an image decoder matching the image encoder to convert the model image from the latent space to the image pixel space. For details, see the attached Figure 2 shown.

[0083] Preferably, the scheme of this embodiment mainly adopts a generation method based on a diffusion model. The diffusion model is a relatively advanced machine learning algorithm that generates high-quality data samples from the noise by gradually adding random noise to the data and then learning the reverse diffusion process. In the generation process, a structurally coherent and detailed image is generated through an iterative approach, repeated diffusion and denoising processes. The diffusion model is a diffusion-based generation model that generates images through multiple stages. At each stage, the model adds some noise to the input noise, thereby gradually changing the content of the image. Through multiple iterations, a high-quality image can eventually be generated. In order to better characterize the generation of high-quality models, a large-scale training data set is constructed to be able to more effectively output a real model effect.

[0084] In addition, GAN (Generative Adversarial Network) can also achieve the purpose of the generated task. GAN is an unsupervised learning model that generates realistic images through adversarial training of a generator network and a discriminator network. The generator network tries to generate images similar to the training data, while the discriminator network tries to distinguish between the images generated by the generator and the real images in the training data. Through this adversarial training, GAN can generate high-quality images. The diffusion model is a diffusion-based generative model that generates images through multiple stages. At each stage, the model adds some noise to the input noise, gradually changing the content of the image. Through multiple iterations, high-quality images can eventually be generated.

[0085] Among them, the generative network requires additional generative control conditions to control the generation of the model image and ensure the controllability of the generation process. The generative control conditions involve model information and clothing information. The overall effect of the clothing is maintained through clothing information, so that the generative network strengthens the understanding and cognition of the local details and global information of digital clothing and digital models, effectively processes the digital model in a targeted manner, and maintains the high fidelity of the design details and elements of digital clothing. Local details and global information are respectively extracted through local area features of the image, such as clothing area, model area, etc., to understand local characteristics; global information is a global understanding of the entire image, such as the matching degree between the model and clothing, the naturalness of the presentation, etc.

[0086] Model information is used to ensure that the newly generated model is consistent with the model in the input image in posture and position. Clothing information specifically refers to detailed information on clothing edges, contours, normals, colors, and textures, while model information involves posture, age, gender, skin color, face shape, facial features, and hairstyle. Any features that can affect the model's image can be included in the model information, effectively enhancing the realism of the virtual model's image. In addition, this embodiment also supports more pre-analysis processing information as generation control conditions, injecting more prior knowledge into generative AI to further improve the controllability and authenticity of generative AI. Here, in order to achieve the controllability of the model, two methods are mainly used to edit the features of posture, age, gender, skin color, face shape, facial features and hairstyle; one method is to use the ability of digital modeling software to predefine digital models that meet the requirements in the process of generating model modeling; the other method is to convert the model configuration item text into feature representation based on the text encoder through configurable model custom input, such as input text fields, and then control and adjust the model generation effect through feature fusion methods, such as feature splicing, cross attention, etc.; the above two methods can be selected or used at the same time. Similarly, clothing information can also be used to generate a reference clothing model, use the information parameters of the model as clothing information, and / or convert the clothing configuration item text into feature representation based on the text encoder through configurable clothing custom input; however, clothing information is not necessary, which depends on whether the user needs to further improve the realism of the clothing through clothing information. For example, in some cases where computing power is insufficient, the user can choose not to input clothing information and only rely on model information to enhance the realism of the model, or the user feels that the clothing also needs to enhance the realism, then the clothing information can also be input into the generation network. This makes the system more interactive and allows users to freely choose the input method of control conditions. If the user has his own model or wants to edit a model of his own, the system provides the function of using the parameters of the model as control conditions, so that the user has more control over the details of the generated network; if the user wants to customize the input through the model, there is no need to edit the model in advance, which is easier for the user to operate and can achieve the desired effect more quickly;

[0087] At the same time, if the user has high requirements for the authenticity and refinement of some human body details, but not for other human body details, the above two methods can be used at the same time to control the generative network. For example, the user has a special personalized design for the tattoo on the model's left arm, and hopes that the image output by the generative network will be retained as it is, while the tattoo on the model's right arm does not have special requirements, and only needs a tattoo to provide a visual effect. In this way, the user can input the model's left arm tattoo into the generative network through model parameters, and input the model's right arm tattoo in the form of text. In this way, on the model rendering processed by the generative network, the tattoo on the model's left arm will be retained as it is, while the right arm will generate a tattoo that roughly meets the user's expectations; the same is true for the enhanced sense of reality of clothing.

[0088] This embodiment also discloses another method for enhancing the sense of reality of a virtual model based on a generation network, the process is as shown in the attached figure. Figure 3 As shown, the specific steps include:

[0089] 301. Receiving a rendering of a virtual model wearing digital clothing;

[0090] 302. Extracting parsed information of the rendering, where the parsed information includes a clothing area and a model area;

[0091] 303. Input the parsed information and the model information into a generation network to generate a model image wearing the digital clothing and corresponding to the model information.

[0092] In this embodiment, the model area and the clothing area are first distinguished and then input into the generative network, so that the generative network can have different processing methods for these two different areas, making the way the entire system processes images more flexible, especially when computing power is limited, the system can more easily invest computing resources in specific areas; at the same time, the generative network can adopt different enhancement strategies for different areas, which will make the enhancement effect better.

[0093] In some specific embodiments, the clothing area is the area where the digital clothing is located, and the model area is the human body structure of the virtual model exposed outside the digital clothing. The model area does not include the human body structure covered by the digital clothing. In some specific embodiments, the model area includes regional contour information and human body part information. Further requirements are made for the separation of the model area and the clothing area. The model area in this embodiment only needs to obtain the structures of each human body part exposed outside the clothing, so that the generation network can selectively process the model area and the clothing area separately. Specifically, the present embodiment can support the user to choose to enhance the sense of reality of the model area or the clothing area, or to enhance both, and under the default preset environment, the system can also select the object of sense of reality enhancement according to the computing resources that the current system can mobilize. For example, in the present embodiment, if the computing resources are not enough to enhance the sense of reality of the model area and the clothing area at the same time, then the system will give priority to enhancing the sense of reality of the model area, and then the user can choose whether to continue to enhance the sense of reality of the clothing; in some cases, the present system can also provide more detailed configuration options, support users to input different detail information for the model area and the clothing area, and control the generation network to enhance the model area and the clothing area respectively. For example, the user can input more requirements for the sense of reality enhancement of the model area, such as details such as tattoos, skin color, scars, etc., and only input texture details for the clothing area. In this way, the present system only realizes the generation of details such as tattoos, skin color, scars, etc. for the model area, and the clothing area only processes the enhancement of texture, and under the system of the present embodiment, the details belonging to the model area will not be generated in the clothing area in a disordered manner.

[0094] In some specific embodiments, before obtaining the analytical information of the rendering, it also includes: determining whether the rendering contains analytical information; if it does not contain analytical information, extracting the clothing area and the model area based on a preset algorithm; and accurately expressing the edges of the clothing area and the model area.

[0095] In some specific embodiments, the clothing area is extracted based on the image segmentation algorithm, the model area is extracted based on the human body analysis algorithm, and the edges of the clothing area and the model area are processed based on the image cutout algorithm to achieve accurate expression of the edges of the clothing area and the model area. Three different schemes are selected for the details involved in clothing, models, and human body structures, so that the expression of details is better. This scheme is to accurately disassemble the image and strengthen the details to complete a high-fidelity try-on rendering, and the inaccurate recognition of human body structure cannot meet the market business needs. From another perspective, it is precisely because of the need for high fidelity that the system needs to invest limited computing resources in more important places. There is no such problem in the field of rendering such as the game industry. In addition, the image cutout algorithm can ensure the integrity of the image and make the visual effect of the whole picture better, while this step is not required for optimizing only the face, because there is no need to consider the impact of clothing on the face, nor is there any need to consider how to enhance the texture and personality of other parts.

[0096] In some specific embodiments, the method of obtaining model information includes: generating a reference model model, using the information parameters of the model as model information; and / or, through a configurable model custom input, converting the model configuration item text into a feature representation through a text encoder, and using the feature representation of the model as model information. Model information is the core generation element of the model, supporting user modeling customization or semantic input. As clothing information is a reference for providing global information, only edge information is considered.

[0097] In some specific embodiments, when the parsing information and the preset model information are input into the generation network, it also includes: inputting the clothing information into the generation network, and the clothing information includes one or more features of the clothing's edge, contour, normal, color, and texture.

[0098] This embodiment provides a method for enhancing the realism of a virtual model based on a generative network. Based on highly controllable generative AI technology, it makes full use of the prior information of digital clothing and virtual models, can significantly improve the realism of clothing models, while maintaining high fidelity of modeled clothing, and is highly adaptable to the alignment of virtual models in the digital world to real-life models in the real world.

[0099] The disclosed embodiment also discloses a virtual model reality enhancement system based on a generative network, which systematizes the virtual model reality enhancement method based on a generative network in the above embodiment to make it more practical. The overall structure diagram of the virtual model reality enhancement system based on a generative network is shown in the attached manual. Figure 4 The specific plan is as follows:

[0100] A virtual model reality enhancement system based on a generative network includes the following modules:

[0101] An input unit 1 is used to obtain a rendering of a virtual model wearing digital clothing;

[0102] The analytical information acquisition unit 2 is used to determine whether the analytical information about the digital clothing in the rendering image is known, and the analytical information includes a clothing area related only to the digital clothing and a model area related only to the human body structure;

[0103] If unknown, the clothing area and model area are extracted based on a preset algorithm;

[0104] A first conversion unit 3, used to convert the parsing information from the image pixel space to the latent space and then input it into the generation network;

[0105] A generating unit 4 is used to enable the generating network to generate a more realistic model image in the model area based on the model information under the control of the generating control conditions including the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing;

[0106] The second conversion unit 5 is used to convert the model image from the latent space to the image pixel space to obtain a generated image of the model wearing the digital clothing.

[0107] Among them, the generation control conditions also include clothing information related to the edges, contours, normals and colors of digital clothing; through clothing information, the generation network strengthens its understanding and cognition of the local details and global information of digital clothing, further ensuring that the model image does not exceed the model area.

[0108] Preferably, the model information includes age, gender, skin color, face shape, facial features and hairstyle; and or, the generative network includes a generative adversarial network and a diffusion model; and / or, the clothing area is segmented based on image segmentation technology, the virtual model area is extracted based on human body analysis technology, and the model area is processed at different granularities to adapt to different scenes based on image cutout technology.

[0109] Preferably, encoding is performed by an image encoder to convert the parsing information from the image pixel space to the latent space; decoding is performed by an image decoder matched with the image encoder to convert the model image from the latent space to the image pixel space.

[0110] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present disclosure. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices in the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario.

[0111] An embodiment of the present specification also provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements any of the above method embodiments by running the executable instructions.

[0112] Figure 5 This is a schematic diagram of a device structure provided by an exemplary embodiment. Figure 5 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0113] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0114] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0116] The embodiments of the present specification also provide a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of any of the above method embodiments are implemented.

[0117] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0118] The present disclosure provides a method and system for enhancing the sense of reality of a virtual model based on a generative network. Based on a highly controllable generative AI technology, the method makes full use of the prior information of digital clothing and virtual models, can significantly improve the sense of reality of clothing models, while maintaining the high fidelity of modeled clothing, and is highly adaptable to the alignment of virtual models in the digital world to real-life models in the real world.

[0119] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0120] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0121] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0122] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0123] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for enhancing the sense of reality of a virtual model based on a generative network, characterized in that: The steps include: Get renderings of virtual models wearing digital clothing; Determining whether analytical information about the digital clothing in the rendering is known, the analytical information including a clothing area related only to the digital clothing and a model area related only to the human body structure; If unknown, the clothing area and model area are extracted based on a preset algorithm; The parsed information is converted from the image pixel space to the latent space and then input into the generation network; Under the control of generation control conditions including model information, the generation network generates a model image with a stronger sense of reality in the model area based on the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing; After converting the model image from latent space to image pixel space, a generated image of the model wearing the digital clothing is obtained.

2. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: The generation control conditions also include clothing information, which includes the edge, outline, normal and color of the digital clothing; The clothing information enables the generation network to strengthen its understanding and cognition of local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

3. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: The model information includes posture, age, gender, skin color, face shape, facial features and hairstyle.

4. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: The generative network includes a generative adversarial network and a diffusion model.

5. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: Encoding by an image encoder to convert the parsed information from an image pixel space to a latent space; The model image is decoded by an image decoder matched with the image encoder to transform the model image from the latent space into the image pixel space.

6. The method for enhancing the sense of reality of a virtual model according to claim 1, characterized in that: The clothing area is segmented based on the image segmentation technology, the virtual model area is extracted based on the human body analysis technology, and the model area is processed with different granularities to adapt to different scenes based on the image cutout technology.

7. A virtual model reality enhancement system based on a generative network, characterized in that: Includes the following modules: An input unit, for obtaining a rendering of a virtual model wearing digital clothing; a parsing information acquisition unit, configured to determine whether parsing information about the digital clothing in the rendering is known, the parsing information including a clothing area related only to the digital clothing and a model area related only to the human body structure; If unknown, the clothing area and model area are extracted based on a preset algorithm; A first conversion unit, used to convert the parsing information from the image pixel space to the latent space and then input it into the generation network; A generating unit, configured to enable the generating network to generate a model image with a stronger sense of reality in the model area based on the model information under the control of generating control conditions including the model information, wherein the model image maintains the form of the virtual model and wears the digital clothing; The second conversion unit is used to convert the model image from the latent space to the image pixel space to obtain a generated image of the model wearing the digital clothing.

8. The virtual model reality enhancement system according to claim 7, characterized in that: The generation control conditions also include clothing information, which includes the edge, outline, normal and color of the digital clothing; The clothing information enables the generation network to strengthen its understanding and cognition of local details and global information of the digital clothing, further ensuring that the model image does not exceed the model area.

9. The virtual model reality enhancement system according to claim 7, characterized in that: The model information includes posture, age, gender, skin color, face shape, facial features and hairstyle; And or, the generative network includes a generative adversarial network and a diffusion model; And / or, the clothing area is segmented based on the image segmentation technology, the virtual model area is extracted based on the human body analysis technology, and the model area is processed with different granularities adapted to different scenes based on the image cutout technology.

10. The virtual model reality enhancement system according to claim 7, characterized in that: Encoding by an image encoder to convert the parsed information from an image pixel space to a latent space; The model image is decoded by an image decoder matched with the image encoder to transform the model image from the latent space into the image pixel space.

11. A method for enhancing the realism of a virtual model based on a generative network, characterized in that: The steps include: receiving a rendering of a virtual model wearing the digital garment; Extracting parsed information of the rendering, the parsed information including a clothing area and a model area; The analysis information and the model information are input into a generation network to generate a model image wearing the digital clothing and corresponding to the model information.

12. The method for enhancing the sense of reality of a virtual model according to claim 11, characterized in that: The clothing area is the area where the digital clothing is located, and the model area is the human body structure of the virtual model exposed outside the digital clothing. The model area does not include the human body structure covered by the digital clothing.

13. The method for enhancing the sense of reality of a virtual model according to claim 11, characterized in that: Before obtaining the parsing information of the rendering, it also includes: Determining whether the rendering contains parsing information; If the analysis information is not included, the clothing area and the model area are extracted based on a preset algorithm, and the edges of the clothing area and the model area are accurately expressed.

14. The method for enhancing the sense of reality of a virtual model according to claim 13, characterized in that: The clothing area is extracted based on an image segmentation algorithm, the model area is extracted based on a human body analysis algorithm, and the edges of the clothing area and the model area are processed based on an image cutout algorithm to achieve accurate expression of the edges of the clothing area and the model area.

15. The method for enhancing the reality of a virtual model according to claim 11, characterized in that: Methods for obtaining the model information include: Generate a reference model and use information parameters of the model as model information; and / or, Through the configurable model custom input, the model configuration item text is converted into feature representation through a text encoder, and the feature representation of the model is used as the model information.

16. The method for enhancing the sense of reality of a virtual model according to claim 11, characterized in that: The model region includes region contour information and human body part information.

17. The method for enhancing the reality of a virtual model according to claim 11, characterized in that: When the analysis information and the preset model information are input into the generation network, the method further includes: inputting clothing information into the generation network, wherein the clothing information includes one or more features of the edge, contour, normal, color, and texture of the clothing.

18. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 11 to 17 by running the executable instructions.

19. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 11 to 17 are implemented.

Images