Virtual image generation method and device, electronic equipment and storage medium

By filtering and redrawing the template images and face features of real-person images in the virtual image feature library, a virtual image that meets the preset requirements is generated, which solves the problems of long time, poor quality and high cost of virtual image generation in the existing technology. It realizes efficient and low-cost virtual image generation.

CN118967930APending Publication Date: 2024-11-15GUANGZHOU QUYAN NETWORK TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411038092.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing virtual image generation methods have problems such as long production time, easy to generate bad hands, violations of content and high operating costs.

Method used

By obtaining the description text input by the user, filter the matching template virtual images from the virtual image feature library based on the pre-trained image generation model, select the face area using the mask frame, and input the implicit features of the face of the real person image. The image generation model is redrawn to generate the target virtual image.

Benefits of technology

It saves time for virtual image generation, avoids the problem of easily generating bad hands and content violations, reduces operating costs, and ensures that the target virtual image meets the preset image requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967930A_ABST
    Figure CN118967930A_ABST
Patent Text Reader

Abstract

The invention discloses a virtual image generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a description text of a required virtual image input by a user; screening out a matched virtual image from a virtual image feature library based on the description text, and taking the matched virtual image as a template virtual image; selecting a face area to be redrawn from the template virtual image by using the mask frame; a real person picture uploaded by a user is acquired, and the face implicit features of the real person picture are extracted; and inputting the face implicit features and the template virtual image into an image generation model, and redrawing a face region in the template virtual image according to the face implicit features to obtain a target virtual image corresponding to the real person picture. According to the method, the virtual image library composed of various different virtual images is generated in advance, and when the target virtual image is generated, only the face area needs to be redrawn, and the target virtual image does not need to be generated in real time according to a description text, so that the generation time can be saved; the problems of hand damage and content violation can be avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for generating a virtual image. Background Art

[0002] With the continuous development of mobile Internet technology and network communication technology, virtual images, as important carriers of information expression in the virtual world, have been applied in multiple scenarios such as virtual e-commerce anchors, virtual celebrities, and two-dimensional characters, and are showing an expansion trend.

[0003] Using image processing technology to convert real-life images into virtual images often affects the conversion effect due to inadequate control of the detailed features of the virtual images. How to make the features of the generated virtual images consistent with those of the real-life images is an issue that technical personnel in this field need to focus on.

[0004] At present, virtual image generation mainly uses CG (Computer Graphics) technology modeling. If personalization is to be achieved, it is necessary to modify it from the modeling stage. The production time is long, the labor cost is high, and the equipment hardware requirements are high. In addition, the generation of virtual images in various postures is prone to bad hands and may also involve content violations and other issues. It is necessary to connect to content security review, which increases the time and operating costs of generating virtual images. Summary of the invention

[0005] The present invention provides a virtual image generation method to solve the problems of long production time, easy generation of bad hands, illegal content and high operating cost in the existing virtual image generation method.

[0006] In a first aspect, the present invention provides a method for generating a virtual image, comprising:

[0007] Obtaining the description text of the desired virtual image input by the user;

[0008] Based on the description text, a matching virtual image is selected from a virtual image feature library as a template virtual image, wherein the virtual image in the virtual image feature library is an image that meets preset image requirements and is obtained according to a pre-trained image generation model;

[0009] Using a mask frame to select a face area to be redrawn in the template virtual image;

[0010] Obtaining real-person pictures uploaded by users and extracting implicit facial features of the real-person pictures;

[0011] The implicit features of the human face and the template virtual image are input into the image generation model to redraw the face area in the template virtual image according to the implicit features of the human face, so as to obtain a target virtual image corresponding to the real person picture.

[0012] In a second aspect, the present invention provides a virtual image generation device, comprising:

[0013] A description text acquisition module, used to acquire the description text of the desired virtual image input by the user;

[0014] A template virtual image matching module is used to select a matching virtual image from a virtual image feature library based on the description text as a template virtual image, wherein the virtual image in the virtual image feature library is an image that meets the preset image requirements and is obtained according to a pre-trained image generation model;

[0015] A face region selection module, used to select a face region to be redrawn in the template virtual image using a mask frame;

[0016] A facial implicit feature extraction module is used to obtain real-person pictures uploaded by users and extract facial implicit features of the real-person pictures;

[0017] The target virtual image generation module is used to input the implicit features of the face and the template virtual image into the image generation model to redraw the face area in the template virtual image according to the implicit features of the face to obtain a target virtual image corresponding to the real person picture.

[0018] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the virtual image generation method described in the first aspect of the present invention.

[0022] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the virtual image generation method described in the first aspect of the present invention when executed.

[0023] The embodiment of the present invention provides a method for generating a virtual image, including: obtaining a description text of a desired virtual image input by a user; selecting a matching virtual image from a virtual image feature library based on the description text as a template virtual image; selecting a face area to be redrawn in the template virtual image using a mask frame; obtaining a real person picture uploaded by a user and extracting implicit face features of the real person picture; inputting the implicit face features and the template virtual image into an image generation model to redraw the face area in the template virtual image according to the implicit face features, and obtaining a target virtual image corresponding to the real person picture. A virtual image library composed of various virtual images is generated in advance. When generating a target virtual image, only the face area needs to be redrawn, and the target virtual image does not need to be generated in real time according to the description text, which can save generation time; in addition, the virtual image in the virtual image feature library is an image that meets the preset image requirements obtained according to the pre-trained image generation model. When generating the target virtual image, the problem of easy generation of bad hands and illegal content can be avoided, and the human body shape of the target virtual image can be guaranteed to meet the preset image requirements, which can reduce the review process and save operating costs.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of a method for generating a virtual image provided by the first embodiment of the present invention;

[0027] Figure 2 is a schematic diagram of a process of generating a target virtual image provided by the first embodiment of the present invention;

[0028] Figure 3 is a flow chart of a method for generating a virtual image provided by Embodiment 2 of the present invention;

[0029] Figure 4 is a schematic diagram of a process of generating a target virtual image provided by the second embodiment of the present invention;

[0030] Figure 5 is a flow chart of a method for obtaining a virtual image in a virtual image feature library provided in Embodiment 3 of the present invention;

[0031] Figure 6 is a structural schematic diagram of a virtual image generating device provided by a fourth embodiment of the present invention;

[0032] Figure 7 It is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] Embodiment 1

[0035] Figure 1 This is a flowchart of a method for generating a virtual image provided by the first embodiment of the present invention. This embodiment is applicable to the case of generating a virtual image. The method can be executed by a virtual image generating device. The virtual image generating device can be implemented in the form of hardware and / or software. The virtual image generating device can be configured in an electronic device. Figure 1 As shown, the virtual image generation method includes:

[0036] S101. Obtain a description text of a desired virtual image input by a user.

[0037] The conversion process of the virtual image is usually carried out on a smart terminal such as a mobile phone or a computer. For example, a mobile phone is provided with an app (Application) for converting the virtual image, and the user can input a description text of the desired virtual image in the app interface (text or voice).

[0038] The desired avatar is the desired avatar of the user, and the description text is a description of the desired avatar's appearance, dress, scene, and other features, which is used to define the overall style of the avatar. For example, the description text may be "feather fan and silk scarf", "Xia Yuhe by the Daming Lake", or "summer, sea, beach", and these description texts are only examples and are not intended to limit the description text.

[0039] S102: Filter out a matching virtual image from a virtual image feature library based on the description text as a template virtual image.

[0040] The virtual images in the virtual image feature library are images that meet preset image requirements obtained based on a pre-trained image generation model.

[0041] The image generation model in the present invention is pre-trained. In an optional embodiment, the training process of the image generation model is as follows: obtain training sample images of the virtual image to be generated and the expected virtual image corresponding to the training sample images; input the training sample images into the initial image generation model to output the actual virtual image corresponding to the training sample images; for each training sample image, calculate the semantic difference between the corresponding expected virtual image and the actual virtual image, obtain the loss value to determine whether the loss value is greater than a preset loss threshold; if so, determine that the actual virtual image corresponding to the training sample image is a positive example image; if not, determine that the actual virtual image corresponding to the training sample image is a negative example image, and calculate the proportion of positive example images to all generated actual virtual images; when the proportion is greater than the preset proportion value, determine that the image generation model training is completed; when the proportion is less than or equal to the preset proportion value, adjust the parameters of the image generation model based on the loss threshold, and return to the step of inputting the training sample image into the initial image generation model to output the actual virtual image corresponding to the training sample image.

[0042] Once the image generation model is trained, it can be used to generate virtual images.

[0043] Based on the description text, matching virtual images are selected from the virtual image feature library. Specifically, the encoders of the respective encoders can be retrieved through the text and pictures. The features calculated are similar in the implicit space to achieve mutual retrieval between pictures and texts (such as Contrastive Language-Image Pre-Training, CLIP). For example, the user's input is "a girl wearing an orange vest and a white skirt", which is encoded into implicit features through the text encoder, and then the implicit features of the virtual image library generated in the previous step are used to calculate the cosine similarity of the two, and the virtual image picture with the highest cosine similarity is returned.

[0044] Among them, the image generation model uses the stable diffusion model (Stable Diffusion) as the initial model, and the stable diffusion model is a kind of deep learning. In this embodiment, the stable diffusion model includes three modules: variational autoencoder (VAE), image / text encoder (Image / Text Encoder) and U-Net. During model reasoning, the image / text encoder encodes the prompt word into the latent vector space, and U-Net obtains the potential representation by denoising from forward diffusion to the reverse direction. The decoder of the variational autoencoder converts the potential representation from the low-dimensional latent space to the pixel space and outputs the final image (in the present invention, it is a virtual image).

[0045] In an optional embodiment, a matching virtual image is selected from a virtual image feature library based on the description text as a template virtual image, including: performing implicit feature extraction on the description text to obtain a first feature; obtaining implicit features of each virtual image in the virtual image feature library as a second feature; and selecting a virtual image from the virtual image feature library based on the matching degree between the first feature and the second feature and using it as a template virtual image.

[0046] Implicit features refer to attributes that have no practical meaning and are difficult to find corresponding features in reality. They are usually embedded features generated by algorithms. The advantage of implicit features is that they have no resolution restrictions, can represent arbitrary topological structures, and have stronger generalization capabilities. For example, in a neural network model, implicit features can be learned through the network. These features are not directly extracted from the data, but are internal representations that the network automatically learns during the training process.

[0047] In an optional example A, a virtual image is selected from a virtual image feature library based on the matching degree of the first feature and the second feature and used as a template virtual image, including: calculating the cosine value similarity between the first feature and the second feature; using the second feature corresponding to the largest cosine value similarity as the target feature with the highest matching degree, and using the virtual image corresponding to the target feature in the virtual image feature library as the template virtual image. In this example, the second feature corresponding to the largest cosine value similarity is used as the target feature with the highest matching degree, so that the corresponding template virtual image can be directly and quickly obtained, and the virtual image in the virtual image feature library meets the preset image requirements, which can avoid problems such as bad hands and content violations, and can ensure the quality of image generation.

[0048] In an optional example B, a virtual image is selected from a virtual image feature library based on the matching degree of the first feature and the second feature and used as a template virtual image, including: calculating the cosine value similarity between the first feature and the second feature; judging whether there is a cosine value similarity greater than a preset cosine value; if so, taking the second feature corresponding to the largest cosine value similarity as the target feature with the highest matching degree, and taking the virtual image corresponding to the target feature in the virtual image feature library as the template virtual image; if not, generating a new virtual image in real time based on the first feature as the template virtual image. In this example, when there is no cosine value similarity greater than the preset cosine value, it means that the implicit features of the virtual image in the virtual image feature library have a low matching degree with the implicit features of the description text. At this time, using the virtual image with the highest matching degree as the template virtual image may be difficult to meet the user's expectations for the virtual image. Therefore, at this time, a new virtual image can be generated in real time based on the first feature as a template virtual image to meet the user's expectations for the virtual image. Generally speaking, after a new virtual image is generated in real time, it needs to be manually reviewed to avoid problems such as bad hands and content violations.

[0049] It should be noted that, in actual applications, different processing methods can be adopted according to actual needs. For example, when it is the peak period of online synthesis, there is a problem of online synthesis peak concurrency, and manual review resources are also relatively tight. In this case, the solution of Example A can be used to obtain the template virtual image, so that the template virtual image can be obtained in time to avoid users waiting for too long. When it is the off-peak period of online synthesis, manual review resources are relatively abundant. The solution of Example B can be used to ensure that the template virtual image meets user expectations and the user waiting time will not be too long.

[0050] S103: Use the mask frame to select the face area to be redrawn in the template virtual image.

[0051] Specifically, a face detection model can be used to perform face detection on the template virtual image, and semantic segmentation can be performed on the face area of ​​the template virtual image to generate a mask map. In the mask map, the face area to be redrawn in the template virtual image is surrounded by a mask frame to distinguish it from other areas.

[0052] The mask can hide or display a specific part of an image without destroying the original image. In the present invention, a mask frame is used to select the face area to be redrawn in the template virtual image, and the face area is displayed, while other areas are hidden. The displayed face area is the part to be redrawn, and the other areas remain unchanged.

[0053] S104: Obtain the real-person picture uploaded by the user and extract implicit facial features of the real-person picture.

[0054] The virtual image conversion process is usually carried out on smart terminals such as mobile phones or computers. The user interface is the interface of the smart terminal, and users can upload real-life pictures locally on the user interface. In addition, the smart terminal may also be equipped with a camera device, and users can use the camera device on the smart terminal to take and upload real-life images in real time.

[0055] The implicit features of the face are the implicit features of the face. The method obtains the real-life picture uploaded by the user and extracts the implicit features of the face of the real-life picture, including: obtaining the real-life picture uploaded by the user; determining whether the real-life picture contains a face; if so, extracting the implicit features of the face from the real-life picture; if not, outputting a prompt message on the user interface to re-upload the real-life picture.

[0056] In the present invention, usually only the face area of ​​the existing template virtual image is redrawn. Therefore, it is necessary to obtain the user's implicit facial features and redraw the face area based on the implicit facial features. When it is detected that the real-life picture does not contain a face, it cannot be used to generate the target virtual image, and the real-life picture is judged to be unqualified, so a prompt message to re-upload the real-life picture is output on the user interface.

[0057] S105, inputting the implicit features of the human face and the template virtual image into the image generation model, so as to redraw the face area in the template virtual image according to the implicit features of the human face, and obtain a target virtual image corresponding to the real person picture.

[0058] Since the template virtual image uses a mask to display the face area and hide other areas, the image generation model can be directly used to redraw the face area. Specifically, the implicit features of the face and the template virtual image are input into the image generation model, and the stable diffusion model is used to perform several steps of redrawing; the facial feature information is added to guide the stable diffusion model to generate a virtual image similar to the user's picture. In addition, the IP-Adapter model can be used while redrawing, so that the implicit features of the face are reflected in the face area of ​​the template virtual image in the form of style transfer, and a target virtual image with better fusion effect is obtained.

[0059] The template virtual image can be matched directly from the virtual image feature library without the need for real-time generation. By using the mask frame to make the face area visible, the face implicit features can be used to redraw the face area to obtain the target virtual image, which saves the real-time generation time of most areas of the target virtual image and improves the generation efficiency. At the same time, it can also ensure that the target virtual image meets the preset image requirements.

[0060] It should be noted that the overall elements of the virtual image include human body shape, character clothing (including style, etc.), skin color, background, scene, etc. Compared with other elements, human body shape is more difficult to draw, takes a long time to draw, and is prone to poor drawing. Therefore, the drawing of human body shape is particularly important. This embodiment focuses on drawing the human body shape part, pre-setting the template virtual image, and only redrawing the face area. This can not only reduce the drawing time of the entire target virtual image, but also fully ensure that the human body shape meets the preset image requirements. The drawing of other elements is usually easier to draw, takes a short time to draw, and is not prone to poor drawing. Therefore, this embodiment does not focus on drawing other elements.

[0061] In order to clearly describe the process of generating the target virtual image, we first combine the following examples T1 and Figure 2 To explain, Figure 2 A schematic diagram of a process for generating a target virtual image is shown in FIG. Figure 2 As shown, the process of generating a target virtual image includes:

[0062] Step 1: The description text X is "a boy wearing a shirt". Based on the description text X, a template virtual image (a) is matched in the virtual image feature library, and the face area of ​​the template virtual image (a) is selected (using a mask frame);

[0063] Step 2: Extract implicit facial features from real-person images (b) uploaded by users;

[0064] In the third step, the template virtual image (a) and the real person image (b) with the selected face area are input into the image generation model to obtain the target virtual image (c).

[0065] It should be noted that, since there is a certain degree of randomness in the generation of the target virtual image, even with the same description text and the same real-life image, the resulting target virtual image may be different, and the present invention is not limited to this.

[0066] The embodiment of the present invention provides a method for generating a virtual image, including: obtaining a description text of a desired virtual image input by a user; selecting a matching virtual image from a virtual image feature library based on the description text as a template virtual image; selecting a face area to be redrawn in the template virtual image using a mask frame; obtaining a real person picture uploaded by a user and extracting implicit features of the face of the real person picture; inputting the implicit features of the face and the template virtual image into an image generation model to redraw the face area in the template virtual image according to the implicit features of the face, and obtain a target virtual image corresponding to the real person picture. A virtual image library composed of various virtual images is generated in advance. When generating a target virtual image, only the face area needs to be redrawn, and the target virtual image does not need to be generated in real time according to the description text, which can save generation time; in addition, the virtual image in the virtual image feature library is an image that meets the preset image requirements obtained according to a pre-trained image generation model. When generating a target virtual image, the problem of easy generation of bad hands and content violations can be avoided, and the review process can be reduced, saving operating costs.

[0067] Embodiment 2

[0068] Figure 3 This is a flow chart of a method for obtaining a virtual image feature library provided in the second embodiment of the present invention. The embodiment of the present invention is further optimized on the basis of the above-mentioned first embodiment. Figure 3 As shown, the method for obtaining the virtual image feature library includes:

[0069] S301, obtaining a description text of a desired virtual image input by a user;

[0070] S302: Filter out a matching virtual image from a virtual image feature library based on the description text as a template virtual image.

[0071] The virtual images in the virtual image feature library are images that meet the preset image requirements obtained based on the pre-trained image generation model;

[0072] S303, using the mask frame to select the face area to be redrawn in the template virtual image;

[0073] S304, obtaining a real person picture uploaded by a user and extracting implicit facial features of the real person picture;

[0074] S305, inputting the implicit features of the human face and the template virtual image into the image generation model, so as to redraw the face area in the template virtual image according to the implicit features of the human face, and obtain a target virtual image corresponding to the real person picture.

[0075] S301-S305 are similar to S101-S105 in the first embodiment, and the details may refer to the relevant contents of the first embodiment, which will not be elaborated here.

[0076] S306: Detect whether the user needs to customize the clothing color and hair color of the target virtual image.

[0077] If yes, then S307 is executed. If no, then no operation is performed.

[0078] S307: Obtain the custom color information input by the user.

[0079] S308, selecting the hair area and clothing area to be colored in the target virtual image through the mask frame.

[0080] The step of selecting the hair area and clothing area to be colored is similar to S103 in the first embodiment (using the mask frame to select the face area to be redrawn in the template virtual image), and the details can be referred to this part.

[0081] S309, combining the preset edge control network and coloring the hair area and clothing area according to the custom color information to obtain an updated target virtual image.

[0082] The edge control network is a control network based on edge detection. The purpose of edge detection is to identify points in a digital image where brightness changes significantly. It is known that the edges of an image are usually points where brightness changes significantly. Therefore, the edge control network can avoid exceeding the hair area and clothing area during recoloring, so that the clothing and hair of the target virtual image before and after the update are similar, with only the color being different, thereby reducing the probability of generating bad images.

[0083] In order to clearly describe the update process of the target virtual image, we first combine the following examples T2 and Figure 4 To explain, Figure 4 is a schematic diagram of a process of updating a target virtual image, such as Figure 4 As shown, the process of updating the target virtual image includes:

[0084] Step 1: Obtain the target virtual image (c). For the specific process, please refer to Example T1 in Embodiment 1.

[0085] Step 2: Select the clothing area and hair area of ​​the target avatar (c);

[0086] In the third step, the clothing area and hair area of ​​the target virtual image (c) are colored according to the user's customized color information to obtain an updated target virtual image (d).

[0087] Through the above steps, the hair area and clothing area of ​​the target virtual image can be colored while ensuring the quality of the target virtual image, thereby meeting the user's customization needs.

[0088] In addition, the background, scene and other elements of the target virtual image may also be modified, which is not limited in the present invention.

[0089] Embodiment 3

[0090] Figure 5 This is a flow chart of a method for obtaining a virtual image in a virtual image feature library provided in a third embodiment of the present invention. Based on the first embodiment above, the present embodiment further refines the process of obtaining a virtual image in a virtual image feature library, such as: Figure 5 As shown, the method for obtaining the virtual image feature library includes:

[0091] S501: Define category prompt words and character posture labels for generating a virtual image.

[0092] Category prompt words may be time, weather, scene, character (upper / lower body) clothing, character accessories, etc. Character posture labels may be standing, sitting, waving hands, tilting head, etc.

[0093] Optionally, after defining the category prompt words and character posture labels for generating the virtual image, the method further includes: for each category prompt word, semantically expanding the category prompt word with the category prompt word as the superordinate word to obtain the subordinate words corresponding to the category prompt word; using the subordinate words corresponding to each category prompt word to perform permutations and combinations to obtain combined prompt words, each combined prompt word including at least one subordinate word; and using the combined prompt word as a new category prompt word.

[0094] For example, the subordinate words of "time" may include morning, noon, afternoon, night, etc., the subordinate words of "weather" may include rainy day, sunny day, foggy day, snowy day, etc., the subordinate words of "character clothing" may include shirt, skirt, cotton-padded jacket, etc., and the subordinate words of "scene" may include woods, wooden house, stream, etc. Exemplarily, the combined prompt words may be "morning, rainy day, woods, skirt", or "morning & sunny day & woods & shirt". In the combined prompt words, different subordinate words may be separated by a variety of preset placeholders, and the present invention does not limit the form of the placeholders.

[0095] By arranging and combining the subordinate words corresponding to each category prompt word, combined prompt words under different combinations of time, weather, scenes, etc. can be obtained, which can enrich the category prompt words, that is, the types and styles of virtual images can be enriched.

[0096] S502: Acquire multiple reference images corresponding to character posture labels.

[0097] Specifically, character posture pictures corresponding to character posture labels are collected as posture references for generating virtual images, that is, character posture pictures are reference pictures, which can make the virtual images varied and avoid character movements being too single. It should be noted that these reference pictures are also pictures that meet the preset image requirements, and manual screening can be used to ensure that the reference pictures meet the preset image requirements.

[0098] S503: Input the category prompt words and reference pictures into a pre-trained image generation model to generate a virtual image of the current round.

[0099] S504: Screen the virtual images of the current round according to preset image requirements to obtain qualified virtual images and unqualified virtual images.

[0100] The screening method of this step can be manual screening to ensure that unqualified virtual images can be accurately identified and the pass rate can be improved.

[0101] S505: Add the qualified virtual images of the current round to the virtual image feature library.

[0102] The qualified virtual images of the current round are obtained through screening and meet the image requirements, so the qualified virtual images of the current round can be added to the virtual image feature library.

[0103] S506: Count the pass rates of the category prompt words and character posture labels corresponding to the qualified virtual images and the unqualified virtual images.

[0104] Even different virtual images generated by the same category prompt word or character posture label may be partially qualified and partially unqualified. Therefore, the pass rate of the category prompt words and character posture labels corresponding to each virtual image (including qualified virtual images and unqualified virtual images) needs to be counted. For example, the category prompt words "dusk, blue hair and shirt" correspond to virtual images A1 and A2. Virtual image A1 is an unqualified virtual image. At this time, the category prompt word is determined to be unqualified, and the number of unqualified is +1. Virtual image A2 is a qualified virtual image. At this time, the category prompt word is determined to be unqualified, and the number of qualified is +1.

[0105] S507: Determine whether the pass rates of the category prompt words and the character posture labels meet the preset pass rate standards.

[0106] If yes, the process returns to S502 , and if no, the process returns to S508 .

[0107] The pass rate refers to the ratio of the number of qualified words to the total number. Category prompt words and character posture labels are of different types, so the pass rates of category prompt words and character posture labels are calculated separately.

[0108] The preset pass rate standard may be a unified standard, such as 90%; or pass rate thresholds may be set for category prompt words and person posture labels respectively.

[0109] For example, the preset pass rate standard can be:

[0110] The pass rate threshold of the category prompt words is set to 95%, and when the pass rate of the category prompt words is greater than or equal to 95%, it is determined that the pass rate of the category prompt words meets the preset pass rate standard;

[0111] The pass rate threshold of the character posture label is 90%. When the pass rate of the character posture label is greater than or equal to 95%, it is determined that the pass rate of the character posture label meets the preset pass rate standard.

[0112] When both meet the preset pass rate standard, the process returns to S502 to generate the next round of virtual images; when at least one of the two does not meet the preset pass rate standard, the process executes S508 to update (screen) the category prompt words and character posture labels.

[0113] S508: Delete the category prompt words and character posture labels whose pass rates do not meet the preset pass rate standard to update the category prompt words and character posture labels.

[0114] That is, the prompt words and character posture labels that are likely to generate bad images (unqualified virtual images) can be excluded, and the prompt words and character posture labels that are likely to generate good images (qualified virtual images) can be obtained. After executing S508, the process returns to execute S502.

[0115] In this embodiment, by setting the pass rate to screen the category prompt words and character posture labels, the probability of generating good images can be increased and the probability of generating bad images can be reduced, so that the generated virtual image can meet the preset image requirements, so as to promote multiple rounds of large-scale virtual image generation and improve the generation efficiency and quality of virtual images.

[0116] It should be noted that, since the process of generating the virtual image is through multiple rounds of image generation and screening and the updating of the category prompt words and character posture labels, when it is not determined that the current category prompt words and character posture labels can generate a virtual image that meets the image requirements, a single batch of reference pictures is temporarily used in sequence to generate the virtual image, and each batch of generated virtual images is screened. After multiple rounds of screening, it is determined that the current category prompt words and character posture labels can generate a virtual image that meets the image requirements and after multiple rounds of verification, it is not necessary to screen the qualified virtual images, the category prompt words and character posture labels, and directly implement S501-S503, and then add all the generated virtual images to the virtual image library.

[0117] In order to clearly illustrate the screening and verification process, illustratively, the screening round is N, the initial value of N is 1, the verification round is M, and the verification round threshold is 2.

[0118] First, filter the category prompt words and character posture labels:

[0119] N=1, S501-S508 are implemented, and the pass rate of the category prompt words and the character posture labels does not meet the preset pass rate standard;

[0120] N=2, S501-S508 are implemented, and the pass rate of the category prompt words and the character posture labels does not meet the preset pass rate standard;

[0121] N=3, S501-S508 are implemented, and the pass rate of the category prompt words and the character posture labels meets the preset pass rate standard;

[0122] To verify:

[0123] M=1, S501-S508 are implemented, and the pass rate of the category prompt words and the character posture labels meets the preset pass rate standard;

[0124] M=2, S501-S508 are implemented, and the pass rates of the category prompt words and the character posture labels meet the preset pass rate standards.

[0125] At this time, S501-S503 can be directly implemented, and then all the generated virtual images are added to the virtual image library. If the pass rate does not meet the preset pass rate standard during the verification, the category prompt words and character posture labels are re-screened.

[0126] Embodiment 4

[0127] Figure 6 This is a schematic diagram of the structure of a virtual image generation device provided by Embodiment 4 of the present invention. Figure 6 As shown, the virtual image generating device includes:

[0128] A description text acquisition module 601 is used to acquire a description text of a desired virtual image input by a user;

[0129] A template virtual image matching module 602 is used to select a matching virtual image from a virtual image feature library based on the description text as a template virtual image, wherein the virtual image in the virtual image feature library is an image that meets the preset image requirements and is obtained according to a pre-trained image generation model;

[0130] A face region selection module 603 is used to select a face region to be redrawn in the template virtual image using a mask frame;

[0131] The face implicit feature extraction module 604 is used to obtain the real person picture uploaded by the user and extract the face implicit features of the real person picture;

[0132] The target virtual image generation module 605 is used to input the implicit facial features and the template virtual image into the image generation model to redraw the facial area in the template virtual image according to the implicit facial features to obtain a target virtual image corresponding to the real person picture.

[0133] Optionally, the template virtual image matching module 602 includes:

[0134] A first feature extraction submodule, used for performing implicit feature extraction on the description text to obtain a first feature;

[0135] A second feature extraction submodule is used to obtain implicit features of each virtual image in the virtual image feature library as a second feature;

[0136] The template virtual image matching submodule is used to select a virtual image from the virtual image feature library based on the matching degree between the first feature and the second feature and use it as a template virtual image.

[0137] Optionally, the template virtual image matching submodule includes:

[0138] a cosine value similarity calculation unit, configured to calculate the cosine value similarity between the first feature and the second feature;

[0139] A similarity determination unit, used to determine whether there is a cosine value similarity greater than a preset cosine value; if so, execute the content of the first determination unit, if not, execute the content of the second determination unit;

[0140] A first determining unit is used to use the second feature corresponding to the largest cosine value similarity as a target feature with the highest matching degree, and use the virtual image in the virtual image feature library that corresponds to the target feature as a template virtual image;

[0141] The second determining unit is used to generate a new virtual image in real time based on the first feature as a template virtual image.

[0142] Optionally, the face implicit feature extraction module 604 includes:

[0143] The real-life picture acquisition submodule is used to acquire real-life pictures uploaded by users;

[0144] A submodule, used to determine whether the real person picture contains a human face; if so, execute the content of the face implicit feature extraction submodule; if not, re-upload the content of the prompt submodule;

[0145] A face implicit feature extraction submodule, used to extract face implicit features from the real person picture;

[0146] The re-upload prompt submodule is used to output prompt information for re-uploading the real-person picture on the user interface.

[0147] Optionally, the virtual image generating device further includes:

[0148] A definition module, used to define category prompt words and character posture labels for generating virtual images;

[0149] A reference picture acquisition module, used to acquire a reference picture corresponding to the character posture label;

[0150] A virtual image generation module, used for inputting the category prompt word and the reference image into a pre-trained image generation model to generate a virtual image of the current round;

[0151] A screening module, used to screen the virtual images of the current round according to the preset image requirements to obtain qualified virtual images and unqualified virtual images;

[0152] An adding module, used for adding the qualified virtual image of the current round to the virtual image feature library;

[0153] A pass rate statistics module, used to count the pass rates of the category prompt words and the character posture labels corresponding to the qualified virtual images and the unqualified virtual images;

[0154] The pass rate judgment module is used to judge whether the pass rate of the category prompt words and the character posture labels meets the preset pass rate standard; if so, return to execute the content of the reference image acquisition module; if not, first execute the content of the deletion module, and then return to execute the content of the reference image acquisition module;

[0155] The deletion module is used to delete the category prompt words and the character posture labels whose pass rates do not meet the preset pass rate standards, so as to update the category prompt words and the character posture labels.

[0156] Optionally, the virtual image generating device further includes:

[0157] A semantic expansion module, for performing semantic expansion on each of the category prompt words by taking the category prompt word as a superordinate word to obtain a subordinate word corresponding to the category prompt word;

[0158] A combination prompt word acquisition module, used to arrange and combine the subordinate words corresponding to each of the category prompt words to obtain a combination prompt word, each of which includes at least one subordinate word;

[0159] The category prompt word updating module is used to use the combined prompt word as the new category prompt word.

[0160] Optionally, the virtual image generating device further includes:

[0161] A custom detection module, used to detect whether the user needs to customize the clothing color and hair color of the target virtual image; if so, execute the content of the custom color information acquisition module;

[0162] A custom color information acquisition module is used to obtain the custom color information input by the user;

[0163] A region selection module, used for selecting a hair region and a clothing region to be colored in the target virtual image through a mask frame;

[0164] The coloring module is used to color the hair area and the clothing area according to the custom color information in combination with a preset edge control network to obtain the updated target virtual image.

[0165] The virtual image generation device provided in the embodiment of the present invention can execute the virtual image generation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0166] Embodiment 5

[0167] Figure 7 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0168] like Figure 7 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0169] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0170] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as a virtual image generation method.

[0171] In some embodiments, the avatar generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the avatar generation method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the avatar generation method in any other appropriate manner (e.g., by means of firmware).

[0172] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0174] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0176] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0177] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0178] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0179] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a virtual image, characterized in that: include: Obtaining the description text of the desired virtual image input by the user; Based on the description text, a matching virtual image is selected from a virtual image feature library as a template virtual image, wherein the virtual image in the virtual image feature library is an image that meets preset image requirements and is obtained according to a pre-trained image generation model; Using a mask frame to select a face area to be redrawn in the template virtual image; Obtaining real-person pictures uploaded by users and extracting implicit facial features of the real-person pictures; The implicit features of the human face and the template virtual image are input into the image generation model to redraw the face area in the template virtual image according to the implicit features of the human face, so as to obtain a target virtual image corresponding to the real person picture.

2. The method for generating a virtual image according to claim 1, wherein: The step of selecting a matching virtual image from a virtual image feature library based on the description text as a template virtual image includes: Performing implicit feature extraction on the description text to obtain a first feature; Obtaining implicit features of each virtual image in the virtual image feature library as the second feature; Based on the matching degree between the first feature and the second feature, a virtual image is selected from the virtual image feature library and used as a template virtual image.

3. The method for generating a virtual image according to claim 2, wherein: The selecting a virtual image from a virtual image feature library based on the matching degree between the first feature and the second feature and using the virtual image as a template virtual image includes: Calculating the cosine similarity between the first feature and the second feature; Determine whether there is a cosine value similarity greater than a preset cosine value; If yes, taking the second feature corresponding to the largest cosine value similarity as the target feature with the highest matching degree, and taking the virtual image in the virtual image feature library corresponding to the target feature as the template virtual image; If not, a new virtual image is generated in real time based on the first feature as a template virtual image.

4. The method for generating a virtual image according to claim 1, wherein: The step of obtaining a real person picture uploaded by a user and extracting implicit facial features of the real person picture includes: Get real-person pictures uploaded by users; Determining whether the real person picture contains a human face; If yes, extracting implicit facial features from the real person picture; If not, a prompt message for re-uploading the real person picture is output on the user interface.

5. The method for generating a virtual image according to any one of claims 1 to 4, characterized in that: The virtual image in the virtual image feature library is obtained by the following steps: Define category prompt words and character posture labels for generating virtual images; Obtaining a reference image corresponding to the character posture label; Inputting the category prompt word and the reference image into a pre-trained image generation model to generate a virtual image of the current round; Screening the virtual images of the current round according to the preset image requirements to obtain qualified virtual images and unqualified virtual images; Adding the qualified virtual image of the current round to the virtual image feature library; Counting the pass rates of the category prompt words and the character posture labels corresponding to the qualified virtual images and the unqualified virtual images; Determining whether the pass rates of the category prompt words and the character posture labels meet the preset pass rate standards; If not, deleting the category prompt words and the character posture labels whose pass rates do not meet the preset pass rate standard to update the category prompt words and the character posture labels; returning to the step of obtaining the reference picture corresponding to the character posture label; If yes, return to the step of obtaining a reference image corresponding to the character posture label.

6. The method for generating a virtual image according to claim 5, wherein: After defining the category prompt words and character posture labels for generating the virtual image, the method further includes: For each of the category prompt words, the category prompt word is used as a superordinate word to perform semantic expansion on the category prompt word to obtain a subordinate word corresponding to the category prompt word; Arrange and combine the subordinate words corresponding to the respective category prompt words to obtain combined prompt words, each of which includes at least one subordinate word; The combined prompt word is used as the new category prompt word.

7. The method for generating a virtual image according to any one of claims 1 to 4, characterized in that: After obtaining the target virtual image corresponding to the real person picture, the method further includes: Detecting whether the user needs to customize the clothing color and hair color of the target avatar; If yes, get the custom color information entered by the user; Selecting the hair area and clothing area to be colored in the target virtual image through a mask frame; The hair area and clothing area are colored according to the custom color information in combination with a preset edge control network to obtain an updated target virtual image.

8. A virtual image generating device, characterized in that: include: A description text acquisition module, used to acquire the description text of the desired virtual image input by the user; A template virtual image matching module is used to select a matching virtual image from a virtual image feature library based on the description text as a template virtual image, wherein the virtual image in the virtual image feature library is an image that meets the preset image requirements and is obtained according to a pre-trained image generation model; A face region selection module, used to select a face region to be redrawn in the template virtual image using a mask frame; A facial implicit feature extraction module is used to obtain real-person pictures uploaded by users and extract facial implicit features of the real-person pictures; The target virtual image generation module is used to input the implicit features of the face and the template virtual image into the image generation model to redraw the face area in the template virtual image according to the implicit features of the face to obtain a target virtual image corresponding to the real person picture.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the virtual image generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the virtual image generation method according to any one of claims 1 to 7 when executed.

Citation Information

Cited By

  • Role face pinching method and system based on AI model, electronic equipment and storage medium

    CN119863590A