Virtual human image generation method, device and equipment based on stable diffusion model

Abstract

The invention provides a virtual human image generation method, device and equipment based on a stable diffusion model, and relates to the technical field of generative artificial intelligence. The method comprises the following steps: carrying out background separation processing on a virtual human original image, and outputting a background-free role main body image; performing feature extraction on the role main body image by adopting a preset multi-label image classification model to generate a role feature label; inputting the role feature tag and an original cue word of the virtual human original image into a large language model, filtering clothes and clothes related vocabularies in the original cue word according to a preset cue word project, and generating an original scene cue word; based on the role main body image, extracting a facial feature vector through a face extraction tool; and inputting the original scene cue word, the facial feature vector and the target scene cue word into a preset stable diffusion model for fusion, and generating a virtual human image adaptive to the target scene cue word. The generated virtual human image is high in consistency, image modification is convenient and flexible, and the clothes changing effect is excellent.

Description

Technical Field

[0001] This invention relates to the field of generative artificial intelligence technology, and in particular to a method, apparatus and device for generating virtual human images based on a stable diffusion model. Background Technology

[0002] With the development of technologies such as artificial intelligence and virtual reality, virtual digital humans (or simply virtual humans) are gradually entering our lives as an emerging digital technology product. Traditional methods for generating virtual human images rely on manual modeling and texture modification, which is inefficient and struggles to achieve dynamic adaptation across multiple scenes. Existing AI-generated image technologies have several shortcomings, primarily: single-model generation cannot simultaneously meet the requirements of preserving character features and adapting to scene styles; existing face-swapping technologies (such as DeepFake) are prone to facial feature shifts and lack coordinated control over overall character elements (such as clothing and accessories); changing clothes is difficult, as traditional techniques require recreating clothing models and fitting them onto the virtual human, a cumbersome process that makes it difficult to achieve fast and diverse clothing changes. These problems limit the flexible application of virtual humans in different scenarios and the user experience. Summary of the Invention

[0003] This invention provides a method, apparatus, and device for generating virtual human images based on a stable diffusion model, which solves the problems of poor consistency of virtual human images in different scenarios, cumbersome image modification, and difficulty in changing clothes in the prior art, so as to realize the controllable transformation of virtual human images in multiple dimensions and meet diverse scenario requirements.

[0004] This invention provides a method for generating virtual human images based on a stable diffusion model, comprising:

[0005] Perform background separation processing on the original image of the virtual human to output a background-free main image of the character;

[0006] A preset multi-label image classification model is used to extract features from the main image of the character and generate character feature labels.

[0007] Input the character feature tags and the original prompt words of the original image of the virtual human into the big language model, filter the clothing and clothing-related words in the original prompt words according to the preset prompt word engineering, and generate the original scene prompt words;

[0008] Based on the main image of the character, facial feature vectors are extracted using a face extraction tool;

[0009] The original scene prompts, the facial feature vectors, and the target scene prompts are input into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompts.

[0010] The method comprises the following steps: inputting an original scene prompt word, a face feature vector, and a target scene prompt word into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompt word; and before the fusion, the method further comprises the following steps:

[0011] generating skeleton key point data of the virtual human original image by OpenPose, and / or generating a hand detail control graph by CannyEdge;

[0012] inputting the skeleton key point data and / or the hand detail control graph into a ControlNet module of the preset stable diffusion model as a generation constraint of the preset stable diffusion model.

[0013] The method comprises the following steps: performing background separation processing on a virtual human original image to output a role subject image without a background.

[0014] calling a ComfyUI-Image Remove BG node of a RemoveBG library;

[0015] segmenting the virtual human original image according to preset parameters to output the role subject image.

[0016] The method comprises the following steps: performing feature extraction on the role subject image by using a preset multi-label image classification model to generate a role feature label.

[0017] inputting the role subject image into the loaded preset multi-label image classification model to output a feature label in a preset probability range;

[0018] removing a background-related label in the feature label and retaining a role body feature label to construct a role feature dictionary.

[0019] The method comprises the following steps: extracting a face feature vector from the role subject image by using a face extraction tool.

[0020] detecting and cropping a face region of the role subject image by using MediaPipe;

[0021] encoding the face region into the face feature vector by using a FaceEncoder module of an Apply InstantID tool.

[0022] According to the virtual human image generation method based on the stable diffusion model provided by the application, the face feature vector is extracted based on the role subject image through a face extraction tool based on the role subject image, and the method further comprises:

[0023] A pre-trained adapter of an IPAdapterFaceID tool is loaded to map the face feature vector to a hidden space of a target model;

[0024] The mapping relationship between the face feature vector and the corresponding style of the target model is learned through a cross-attention mechanism.

[0025] According to the virtual human image generation method based on the stable diffusion model provided by the application, the face feature vector, the target scene prompt word, and the original scene prompt word are input into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompt word, and the method comprises:

[0026] In the UNet network of the preset stable diffusion model, the face feature vector, the original scene prompt word, and the target scene prompt word are fused through a cross-attention layer to generate the virtual human image that maintains the consistency of the face features.

[0027] The application also provides a virtual human image generation device based on a stable diffusion model, comprising:

[0028] An output module is configured to perform background separation processing on the virtual human original image and output a role subject image without a background;

[0029] A first generation module is configured to perform feature extraction on the role subject image by using a preset multi-label image classification model and generate a role feature label;

[0030] A second generation module is configured to input the role feature label and an original prompt word of the virtual human original image into a large language model, filter the clothing and costume related words in the original prompt word according to a preset prompt word engineering, and generate an original scene prompt word;

[0031] An extraction module is configured to extract a face feature vector based on the role subject image through a face extraction tool;

[0032] A third generation module is configured to input the original scene prompt word, the face feature vector, and a target scene prompt word into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompt word.

[0033] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for generating a virtual human image based on a stable diffusion model according to any one of the above when executing the program.

[0034] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the method for generating a virtual human image based on a stable diffusion model according to any one of the above.

[0035] The application provides a method, device and equipment for generating a virtual human image based on a stable diffusion model, which separates the original image of the virtual human from the background to ensure that the virtual human image can be accurately separated from the original background for seamless fusion in different scenes; a preset multi-label image classification model is used to extract role feature labels to realize accurate decoupling of the virtual human image, including extraction of key features such as clothes, hairstyle and accessories, to provide a basis for subsequent image customization and scene adaptation; a large language model is used to filter and optimize the clothes prompt words to improve the virtual human dressing effect, realize quick and diversified clothes changing, and eliminate the need for a complex model reconstruction process; a face extraction tool is used to extract a face feature vector to maintain the role face features while realizing style and scene conversion, solving the problem of poor role consistency in the prior art; finally, the original scene prompt words, the face feature vector and the target scene prompt words are input into a preset stable diffusion model for fusion to generate a virtual human image. The virtual human image generated by the application has high consistency, can effectively ensure the consistency of the virtual human image in different scenes through the backstepping word technology, will not be distorted due to scene changes, improves the stability of the virtual human scene switching, is convenient and flexible for image modification, the use of the face extraction tool (such as the Apply InstantID tool and the IPAdapterFaceID tool) enables the virtual human image modification to be adjusted in a large range of style and finely carved in local details, meets the diversified customization needs of users, improves the user experience, and has excellent dressing effect, realizes quick and diversified dressing of the virtual human by using the AI model, eliminates the need for re-making a clothes model, greatly improves the dressing efficiency, and provides more possibilities for the application of the virtual human in different scenes. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without any creative effort.

[0037] Figure 1A flowchart of a virtual human image generation method based on a stable diffusion model is provided by the present application.

[0038] Figure 2 A structural diagram of a virtual human image generation device based on a stable diffusion model is provided by the present application.

[0039] Figure 3 A structural diagram of an electronic device is provided by the present application. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0041] It should be noted that, in the description of the embodiments of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or device comprising the element. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0042] The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" and the like are generally of a kind, and are not limited to the number of objects, for example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / ", generally means that the front and rear associated objects are in a "or" relationship.

[0043] The present application provides a virtual human image generation method, device and equipment based on a stable diffusion model, which dynamically generates virtual human images in multiple scenes based on a stable diffusion (Stable Diffusion) model, and realizes controllable conversion of virtual human images in multiple dimensions such as clothing, posture, background, facial features, etc. through a three-layer architecture of feature decoupling-element reorganization-model adaptation.

[0044] The core process includes:

[0045] Role feature decoupling: extracting role core elements through image backstepping and semantic analysis;

[0046] Element dynamic reorganization: combining multi-modal control modules to realize feature editing and scene adaptation;

[0047] Model consistency migration: preserving role identity features through lightweight adaptation technology.

[0048] The following will be described in conjunction with Figures 1 to 3 The application provides a virtual human image generation method, device and equipment based on a stable diffusion model.

[0049] As Figure 1 The application provides a virtual human image generation method based on a stable diffusion model, which can specifically include the following steps:

[0050] Step 110, performing background separation processing on the virtual human original image to output a background-free role subject image.

[0051] Specifically, the virtual human original image refers to an image including a virtual human image and a background, the virtual human original image is preprocessed first to remove the background in the image, realizing fast separation of the background and the virtual human, and outputting a background-free role subject image. The image can be in PNG, JPG or other formats, and the image resolution can preferably be greater than or equal to 1024*1024.

[0052] In some embodiments, step 110 can include:

[0053] calling the ComfyUI-Image Remove BG node of the RemoveBG library;

[0054] segmenting the virtual human original image according to preset parameters to output the role subject image.

[0055] In this embodiment, the AI-based picture background removal tool RemoveBG is used, the ComfyUI-Image Remove BG node is called for picture processing, the input virtual human original image is segmented according to preset parameters, and the foreground mask and the background-free role subject image are output.

[0056] In some embodiments, the preset parameters can be set as: rem_mode=RMBG-2.0, add_background=none.

[0057] For example, input the concert original image of virtual idol A (with stage background), perform background separation, remove the stage background in the image, and output the virtual idol A image with transparent background.

[0058] Step 120, using a preset multi-label image classification model to extract features of the role subject image, and generating role feature labels.

[0059] Specifically, the input role subject image is decoupled by a preset multi-label image classification model to extract labeled features and generate role feature labels. The role feature labels can include gender, facial features, hairstyle, clothing, accessories, and the like.

[0060] In some embodiments, a multi-label image classification model WD14 Tagger can be used for feature extraction.

[0061] In some embodiments, step 120 can include:

[0062] Inputting the role subject image into the loaded preset multi-label image classification model to output feature labels in a preset probability range;

[0063] Excluding background-related labels in the feature labels and retaining role body feature labels to construct a role feature dictionary.

[0064] In this embodiment, a pre-trained multi-label image classification model (such as the wd-v1-4-moat-tagger-v2 model, and the label library contains more than 6000 categories) is loaded, the role subject image is input, and the feature labels in a preset probability range (such as threshold = 0.30, character_threshold = 0.85) are output. The output feature labels include background-related labels and role body feature labels. Irrelevant labels (such as blackbackground* and upperbody*) are excluded using exclude_tags, and role body feature labels are retained to construct a role feature dictionary as a basic constraint condition for subsequent generation of virtual human images.

[0065] Step 130, inputting the role feature labels and the original prompt words of the virtual human original image into a large language model, filtering clothing and accessories related words in the original prompt words according to a preset prompt word engineering, and generating original scene prompt words.

[0066] Specifically, the clothing description words of the original virtual human role are automatically filtered by using a large language model to improve the dressing effect of the virtual human changing different clothes. The original prompt of the original image of the virtual human and the role ontology feature label are input into a large language model (such as GPT-4), and the large language model is guided to filter the clothing and clothing related words in the original prompt through a preset prompt engineering. The preset prompt engineering includes instructions for removing clothing and clothing related words.

[0067] For example, the API of Doubao-1.5-pro-32k is called, and the following prompt engineering is input:

[0068] "You are a professional and rigorous English translation expert with extensive experience in the international clothing industry, and have a comprehensive and accurate clothing and clothing vocabulary knowledge system. Your task is to carefully read the given paragraph of illogical content, and use your professional knowledge to find all the words and vocabulary related to clothing, clothing, sleeves, etc. in a comprehensive and detailed manner."

[0069] The large language model filters out the clothing and clothing related words in the original prompt, and finally generates the original scene prompt without clothing and clothing related words. For example, "He wears a suit, tie and metal watch to attend a business meeting", and after filtering, it becomes "He goes to attend a business meeting".

[0070] In step 140, the face feature vector is extracted from the role subject image based on the role subject image through a face extraction tool.

[0071] Specifically, the face feature vector in the role subject image is extracted by the face extraction tool, realizing the "face changing" of the virtual human, maintaining the role facial features while realizing the style and scene conversion.

[0072] Wherein, the face changing includes same model face changing and cross model migration. The same model face changing means that under the same stable diffusion model, the role identity is preserved through face feature embedding, and the scene style conversion is realized by combining dynamic prompt words. Cross model migration means migrating the role image between different source models (such as migrating the realistic model to the two-dimensional model), solving the problem of face feature distortion caused by model style difference.

[0073] For the same model face changing, step 140 can include:

[0074] The face region of the role subject image is detected and cropped by MediaPipe;

[0075] The face region is encoded into a face feature vector by the FaceEncoder module of the Apply InstantID tool.

[0076] Specifically, the face swapping with the model can be achieved by the Apply InstantID tool. The face region of the character subject image is cropped by MediaPipe detection, and the face features of the face region are encoded into a latent space vector, i.e., a face feature vector (for example, a 512-dimensional face feature vector), by using the FaceEncoder module of the Apply InstantID, which captures face details such as facial feature ratios, skin colors, and the like. Subsequently, by using a stable diffusion model, the face feature vector is fused with a target scene prompt word (such as “science fiction mecha style”) to generate a virtual human image that maintains facial consistency while adapting to a new scene.

[0077] For cross-model migration, step 140 can further include:

[0078] loading a pre-trained adapter of the IPAdapterFaceID tool to map the face feature vector to the latent space of the target model;

[0079] learning the mapping relationship between the face feature vector and the style of the target model by cross-attention mechanism.

[0080] Specifically, the cross-model migration can be achieved by the IPAdapterFaceID tool, for example, migrating the face features of a virtual human image of a realistic model to a two-dimensional model. When switching models (such as from a realistic model to a two-dimensional model), a pre-trained adapter of the IPAdapterFaceID is loaded, which maps the face feature vector of the source character to the latent space of the target model (two-dimensional model), and learns the mapping relationship between the source character and the style of the target model by cross-attention mechanism. Subsequently, in the generation, the scene adaptation prompt word is combined to ensure that the character identity features (such as facial contours) remain unchanged after the cross-model, while meeting the style characteristics of the target model. For example, by fusing the “two-dimensional bedroom scene” prompt word, a virtual human image in a two-dimensional style is generated.

[0081] Step 150, inputting the original scene prompt word, the face feature vector, and the target scene prompt word into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompt word.

[0082] Specifically, step 150 can include: in the UNet network of the preset stable diffusion model, fusing the face feature vector, the original scene prompt word, and the target scene prompt word by a cross-attention layer to generate the virtual human image that maintains facial feature consistency.

[0083] In the UNet of the stable diffusion model (SDXL 1.5), the facial features are fused by a cross-attention layer to finally generate a virtual person image, which retains the facial features of the virtual person in the original virtual person image. The target scene prompt is fused by using the stable diffusion model, and the virtual person image is quickly and diversely changed without the need to re-produce a clothing model.

[0084] In some embodiments, before step 150, the following can also be included:

[0085] The skeleton key point data of the virtual person original image is generated by OpenPose, and / or the hand detail control map is generated by CannyEdge;

[0086] The skeleton key point data and / or the hand detail control map are input into the ControlNet module of the preset stable diffusion model as the generation constraint of the preset stable diffusion model.

[0087] In this embodiment, OpenPose (control_v11p_sd15_openpose.pth) is used to generate the target pose skeleton map (such as the skeleton map) of the virtual person image, and the JSON coordinate data of multiple key points are output, which are used as the generation constraint. The stable expansion model combines the pose constraint, the scene prompt word, and the facial feature vector to generate the final image of the virtual person (for example, virtual idol A wearing a home clothes and sitting on the sofa in the living room).

[0088] In addition, Canny Edge (control_v11p_sd15_canny.pth) can also be used to generate a hand detail control map to realize hand detail control of the virtual person image (for example, virtual idol A wearing a home clothes and holding a wine glass on the sofa in the living room).

[0089] In some embodiments, the Remove BG tool can also combine the Inpainting module of the stable diffusion model to generate a background that conforms to the target scene lighting logic, thereby realizing the scene replacement of the virtual person image.

[0090] The virtual human image generation method based on the stable diffusion model of the above embodiment can ensure that the virtual human image can be accurately separated from the original background, so as to be seamlessly fused in different scenes; the preset multi-label image classification model is used to extract the role feature label, so as to realize accurate decoupling of the virtual human image, including extraction of key features such as clothes, hairstyle and accessories, and provide a basis for subsequent image customization and scene adaptation; the large language model is used to filter and optimize the clothes prompt word, so as to improve the virtual human dressing effect, realize quick and diversified clothes changing, and avoid a complex model reconstruction process; the face extraction tool is used to extract the face feature vector, so as to maintain the role face features while realizing the conversion of the style and the scene, and solve the problem of poor role consistency in the prior art; finally, the original scene prompt word, the face feature vector and the target scene prompt word are input into the preset stable diffusion model for fusion to generate the virtual human image.

[0091] The generated virtual human has the following advantages:

[0092] 1. High image consistency: through the back-propagation word technology, the virtual human image consistency in different scenes can be effectively ensured, and the virtual human image will not be distorted due to scene changes, thereby improving the stability of the virtual human scene switching;

[0093] 2. Convenient and flexible image modification: the use of the face extraction tool (such as the Apply InstantID tool and the IPAdapterFaceID tool) enables the virtual human image modification to be adjusted in a large range of styles and finely carved in local details, thereby meeting the diversified customization needs of users and improving the user experience;

[0094] 3. Excellent dressing effect: the AI model is used to realize quick and diversified dressing of the virtual human, and the dressing efficiency is greatly improved without the need to re-produce the clothes model, thereby providing more possibilities for the application of the virtual human in different scenes.

[0095] The virtual human image generation device based on the stable diffusion model provided by the present application is described below, and the virtual human image generation device based on the stable diffusion model described below can be correspondingly referred to the virtual human image generation method based on the stable diffusion model described above.

[0096] As shown in Figure 2 The present application also provides a virtual human image generation device based on a stable diffusion model, which comprises:

[0097] An output module 210 is configured to perform background separation processing on the virtual human original image, and output a role subject image without background;

[0098] A first generation module 220 is configured to perform feature extraction on the role subject image by using a preset multi-label image classification model, and generate a role feature label.

[0099] The second generation module 230 is configured to input the role feature label and original prompt words of the virtual human original image into a large language model, filter clothing and costume related words in the original prompt words according to a preset prompt word engineering, and generate original scene prompt words.

[0100] The extraction module 240 is configured to extract a face feature vector based on the role subject image through a face extraction tool.

[0101] The third generation module 250 is configured to input the original scene prompt words, the face feature vector, and target scene prompt words into a preset stable diffusion model for fusion, and generate a virtual human image adapted to the target scene prompt words.

[0102] Figure 3 An example of an entity structure diagram of an electronic device is shown in Figure 3 As shown, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute a virtual human image generation method based on a stable diffusion model, which includes:

[0103] performing background separation processing on a virtual human original image to output a role subject image without background;

[0104] adopting a preset multi-label image classification model to extract features of the role subject image to generate a role feature label;

[0105] inputting the role feature label and original prompt words of the virtual human original image into a large language model, filtering clothing and costume related words in the original prompt words according to a preset prompt word engineering, and generating original scene prompt words;

[0106] extracting a face feature vector based on the role subject image through a face extraction tool;

[0107] inputting the original scene prompt words, the face feature vector, and target scene prompt words into a preset stable diffusion model for fusion, and generating a virtual human image adapted to the target scene prompt words.

[0108] In addition, the logic instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0109] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the virtual human image generation method based on the stable diffusion model provided by the above-mentioned methods, the method comprising:

[0110] performing background separation processing on the virtual human original image to output a role subject image without background;

[0111] adopting a preset multi-label image classification model to perform feature extraction on the role subject image to generate a role feature label;

[0112] inputting the role feature label and the original prompt word of the virtual human original image into a large language model, filtering the clothing and costume related words in the original prompt word according to a preset prompt word engineering, and generating an original scene prompt word;

[0113] extracting a face feature vector based on the role subject image through a face extraction tool;

[0114] inputting the original scene prompt word, the face feature vector, and a target scene prompt word into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompt word.

[0115] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, the computer program is executed by a processor to implement the virtual human image generation method based on the stable diffusion model provided by the above-mentioned methods, the method comprising:

[0116] performing background separation processing on the virtual human original image to output a role subject image without background;

[0117] characteristic label of the role is generated by using a preset multi-label image classification model to perform feature extraction on the role subject image;

[0118] The role characteristic label and the original prompt word of the virtual human original image are input into a large language model, and a preset prompt word engineering is performed on the original prompt word to filter out clothing and costume related words to generate an original scene prompt word;

[0119] Based on the role subject image, a face feature vector is extracted by a face extraction tool;

[0120] The original scene prompt word, the face feature vector, and the target scene prompt word are input into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompt word.

[0121] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0122] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0123] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating virtual human images based on a stable diffusion model, characterized in that, include: Perform background separation processing on the original image of the virtual human to output a background-free main image of the character; A preset multi-label image classification model is used to extract features from the main image of the character and generate character feature labels. Input the character feature tags and the original prompt words of the original image of the virtual human into the big language model, filter the clothing and clothing-related words in the original prompt words according to the preset prompt word engineering, and generate the original scene prompt words; Based on the main image of the character, facial feature vectors are extracted using a face extraction tool; The original scene prompts, the facial feature vectors, and the target scene prompts are input into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompts.

2. The virtual human image generation method based on a stable diffusion model according to claim 1, characterized in that, Before inputting the original scene prompts, the facial feature vectors, and the target scene prompts into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompts, the process further includes: The skeleton key point data of the original image of the virtual human is generated by OpenPose, and / or the hand detail control map is generated by Canny Edge; The skeleton key point data and / or the hand detail control map are input into the ControlNet module of the preset stable diffusion model as generation constraints for the preset stable diffusion model.

3. The virtual human image generation method based on a stable diffusion model according to claim 1, characterized in that, The process of performing background separation processing on the original image of the virtual human to output a background-free main image of the character includes: Call the ComfyUI-Image Remove BG node from the RemoveBG library; The original image of the virtual human is segmented according to preset parameters, and the main image of the character is output.

4. The virtual human image generation method based on a stable diffusion model according to claim 1, characterized in that, The step of extracting features from the main image of the character using a preset multi-label image classification model to generate character feature labels includes: The main image of the character is input into the loaded preset multi-label image classification model, and feature labels with a preset probability range are output. Background-related tags are removed from the feature tags, and the character's own feature tags are retained to construct a character feature dictionary.

5. The virtual human image generation method based on a stable diffusion model according to claim 1, characterized in that, The step of extracting facial feature vectors based on the main image of the character using a face extraction tool includes: MediaPipe was used to detect and crop the facial region of the main image of the character. The face region is encoded into a facial feature vector using the FaceEncoder module of the Apply InstantID tool.

6. The virtual human image generation method based on a stable diffusion model according to claim 5, characterized in that, The step of extracting facial feature vectors based on the main image of the character using a face extraction tool also includes: Load the pre-trained adapter of the IPAdapterFaceID tool to map the facial feature vectors to the latent space of the target model; The mapping relationship between the facial feature vectors and the corresponding style of the target model is learned through a cross-attention mechanism.

7. The virtual human image generation method based on a stable diffusion model according to claim 5 or 6, characterized in that, The step of inputting the original scene prompts, the facial feature vectors, and the target scene prompts into a preset stable diffusion model for fusion to generate a virtual human image adapted to the target scene prompts includes: In the UNet network with the preset stable diffusion model, the facial feature vector, the original scene cue words, and the target scene cue words are fused through a cross-attention layer to generate the virtual human image that maintains the consistency of facial features.

8. A virtual human image generation device based on a stable diffusion model, characterized in that, include: The output module is used to perform background separation processing on the original image of the virtual human and output a background-free image of the main character. The first generation module is used to extract features from the main image of the character using a preset multi-label image classification model and generate character feature labels. The second generation module is used to input the character feature tags and the original prompt words of the original image of the virtual human into the large language model, filter the clothing and clothing-related words in the original prompt words according to the preset prompt word engineering, and generate the original scene prompt words; The extraction module is used to extract facial feature vectors based on the main image of the character using a face extraction tool; The third generation module is used to input the original scene prompts, the facial feature vectors, and the target scene prompts into a preset stable diffusion model for fusion, and generate a virtual human image that is adapted to the target scene prompts.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the virtual human image generation method based on the stable diffusion model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the virtual human image generation method based on the stable diffusion model as described in any one of claims 1 to 7.

Images