Figure style image generation method, system and device based on model training and image-text generation technology, and medium

Abstract

The invention discloses a figure style image generation method, system and device based on model training and image-text generation technologies, and a medium. The method comprises the steps of training an optimized style bottom model, expanding a style bottom model training sample, improving the generalization ability of the style bottom model, and obtaining an optimized target style basic model; the method comprises the following steps: acquiring character image data of a user, preprocessing the character image data, extracting face feature information of the user, and proofreading the face feature information through a user feedback mechanism to obtain final character image training data; inputting the character image training data into a Stable Diffusion model, and training to obtain a character specific LoRA model; setting character style image generation information, and configuring corresponding character style image generation parameters based on the character style image generation information; and on the basis of the target style basic model, the character specific LoRA model and the character style image generation parameters, performing text-to-image processing and image-to-image processing, and finally obtaining a target character style image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, device and medium for generating a character style image based on model training and image-text generation technology. Background Art

[0002] At present, in the existing technology for user portraits and style training images, the solutions that have been disclosed include: using the LoRA training strategy with parameter adaptive budget allocation to enhance the network learning ability under limited training resources, providing a basis for the similarity and authenticity of virtual portraits; introducing facial feature information, inputting the diffusion model in the form of conditional features to assist in portrait generation, and improving the similarity of characters; generating style portraits in a hierarchical and controllable manner, decoupling ID and style, and further decomposing the fine-grained features of the style based on the style LoRA model to achieve controllable portrait style. However, these existing technical solutions still have the following problems:

[0003] The authenticity of the character image synthesis results of the existing technical solutions is insufficient. When extracting facial feature information, it depends on the quality of the image uploaded by the user and the accuracy of the image encoder. There are also many solutions that propose to first tune or optimize the character image to improve the accuracy of the facial feature information and the authenticity of the character image synthesis results. However, these treatments will lead to a certain gap between the synthesized effect of the character image and the real effect. For example, the common problem that the age of the characters in the synthesized image is relatively young / young, and the characters in the synthesized image have heavy makeup. The existing technical solutions have randomness in the character posture when synthesizing character style images. Since the character posture in the current character photo synthesis is mostly controlled based on the prompt word prompt, this will cause a certain randomness in the character posture in the synthesized image. Sometimes the synthesized posture does not meet the user's requirements or the synthesis effect is poor. The existing technical solutions collect and obtain user image data in a relatively simple way, mostly requiring users to provide original images or photos, and then perform image processing and character style image synthesis. Summary of the invention

[0004] The purpose of the present invention is to provide a method, system, device and medium for generating character style images based on model training and image-text generation technology, so as to improve the synthesis effect of character style images and enhance the authenticity of the synthesis results.

[0005] To achieve the above purpose, the technical solution provided by the present invention is:

[0006] The first aspect of the present application provides a method for generating a character style image based on model training and image-text generation technology, comprising the following steps:

[0007] Train and optimize the style base model, expand the style base model training samples, improve the generalization ability of the style base model, and obtain the optimized target style basic model;

[0008] Obtain the user's character image data, pre-process the character image data, extract the user's facial feature information, proofread the facial feature information through the user feedback mechanism, and obtain the final character image training data;

[0009] Input the character image training data into the Stable Diffusion model to train the character-specific LoRA model;

[0010] Setting character style image generation information, and configuring corresponding character style image generation parameters based on the character style image generation information;

[0011] Based on the target style basic model, the character-specific LoRA model and the character style image generation parameters, text-generated images and image-generated images are processed to finally obtain the target character style image.

[0012] The obtaining of the user's character image data is specifically as follows: using the AIGC model tool to generate different character image data, the character images of the character image data cover samples of all age groups and genders, fully learning facial features and expressions, combining multi-scale feature fusion technology and attention mechanism, and strengthening the learning of detailed features and important features of the face in the character image data.

[0013] The pre-processing of the character image data includes non-real image filtering, facial key point detection, image face rotation correction and image beauty style removal processing;

[0014] The facial feature information of the user is extracted and proofread through a user feedback mechanism, specifically: facial feature information is extracted based on a facial feature extraction algorithm, and through a user feedback mechanism, the user is allowed to confirm whether there are errors in the facial feature information extracted by the algorithm, and the user is allowed to modify and adjust.

[0015] To optimize the above technical solutions, the specific measures taken also include:

[0016] The specific process of preprocessing the character image data is as follows:

[0017] (1) Use algorithms or image editing software to remove the beauty effects in the image;

[0018] (2) Use face detection algorithms and face key point detection algorithms to extract faces from images and locate face areas, filter out data that does not contain faces, is not a real face, or has faces that are obscured; calculate the area ratio of faces in the image and filter out data where the face ratio is too small; for images containing multiple faces, use a user feedback mechanism to let users confirm the characters that need to be retained;

[0019] (3) using an image face rotation algorithm, the face data obtained in step (2) is rotated and aligned so that the line connecting the center coordinates of the left and right eyes of the face is in a horizontal state after processing;

[0020] (4) Encoding information on the facial image processed by steps (1) to (3): first, encoding information on the facial image using an image-based text encoder and decoder; then, allowing the user to confirm the information encoded data through a user feedback mechanism; and finally, inputting the information encoded data and the facial image into a portrait base model to train a user-specific LoRA model.

[0021] Furthermore, if the user provides image data, facial feature information in the image is directly extracted after image preprocessing; if the user provides multiple-frame images or dynamic videos, the multiple-frame images or dynamic videos are first converted into an image frame set, each frame image in the image frame set is preprocessed, and then the facial feature information of the target person in the image is extracted.

[0022] Furthermore, the setting of character style image generation information and configuring corresponding character style image generation parameters based on the character style image generation information includes:

[0023] Set the target style type and number of generated images of the character style image, and obtain the associated target style model id and target style prompt word based on them;

[0024] Set the reference action of the character style image and convert it into the corresponding ControlNet character pose parameters.

[0025] Furthermore, the target style basic model, the character-specific LoRA model and the character style image generation parameters are used to perform text-generated image and image-generated image processing, and finally obtain the target character style image. The specific process is as follows:

[0026] (1) Obtain the target style and the number of synthesized character style images selected by the user;

[0027] (2) determining target style synthesis parameters based on the target style selected by the user, including a target style basic model and target style prompt word parameters;

[0028] (3) Obtain the posture reference image selected by the user, determine the posture skeleton image based on the posture reference image, and finally determine the target style ControlNet posture control parameters;

[0029] (4) performing image processing based on the target style synthesis parameters, the posture control parameters, and the user-specific LoRA model to generate an initial version of the character style image set, and then performing image-face nesting fusion processing on the face area in the initial version of the character style image set and any face data used for training the user-specific LoRA model to obtain an enhanced version of the character style image set;

[0030] (5) Calculate the similarity between the facial part of the image in the enhanced version of the character style image set and the facial part of any character image data uploaded by the user that meets the specifications, and sort the similarity results in order from large to small, and output the top n character style images with the highest similarity scores to the user, that is, the final character style image; where n is the number of synthesized character style images selected by the user.

[0031] The second aspect of the present application provides a character style image generation system based on model training and image-text generation technology, including:

[0032] The target style base model optimization module is used to train and optimize the style base model, expand the style base model training samples, improve the generalization ability of the style base model, and obtain the optimized target style base model;

[0033] The character image training data acquisition module is used to obtain the character image data of the user, pre-process the character image data, extract the user's facial feature information, proofread the facial feature information through the user feedback mechanism, and obtain the final character image training data;

[0034] The LoRA model training module is used to input character image training data into the Stable Diffusion model to train a character-specific LoRA model;

[0035] A parameter configuration module, used to set character style image generation information and configure corresponding character style image generation parameters based on the character style image generation information;

[0036] The character style image generation module is used to perform text-generated image and image-generated image processing based on the target style basic model, the character-specific LoRA model and the character style image generation parameters, and finally obtain the target character style image.

[0037] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for generating a character style image based on model training and text-to-image technology as described above is implemented.

[0038] The fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for generating a character style image based on model training and image-text generation technology as described above.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention starts from two aspects: optimizing the style base model training and optimizing the character feature information extraction method. First, the number and diversity of training samples of the style base model are increased, the age groups are subdivided, and image data of people of different age groups are collected based on the natural advantages of operators. At the same time, the AIGC model tool is used to generate image data of people of different age groups, expand the style base model training samples, improve the generalization ability of the style base model, enhance the model diversity, and improve the details of character style image synthesis. Combined with multi-scale feature fusion technology and attention mechanism, the former focuses on different details for images of different scales, and the latter automatically learns and determines the important features in the image to enhance the trained model's attention to facial details and improve the quality and authenticity of the generated character style images. In addition, the image editing software is used to remove the beauty effect in the photo to ensure that the extracted character feature information is as close to the original state as possible, and a user feedback mechanism is established for the facial feature information extracted by the algorithm, so as to enhance the accuracy of character feature information extraction, improve the character style image synthesis effect, and further enhance the authenticity of the synthesis result.

[0041] The present invention proposes a method for fusing posture prompt words with the ControlNet model to assist in controlling the posture of a character in a synthesized character style image, further improve the synthesis effect of the character style image, and reduce the randomness of the character posture in the synthesized character style image.

[0042] The present invention proposes a user multimodal data collection method, allowing users to upload personal pictures, multi-frame images or dynamic videos containing user facial information, including but not limited to user data in PNG, JPEG, JPG, GIF, MP4 and other formats. For multi-frame images and dynamic video data, visual information decoding based on the time dimension is used to realize event sequence analysis of target facial features to obtain more key feature information of the person. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 : A schematic diagram of the process structure of the character style image generation method based on model training and image-text generation technology of the present invention.

[0044] Figure 2 : A flowchart of a specific implementation method of the pre-steps in the character style image generation method based on model training and text-to-image technology of the present invention.

[0045] Figure 3: The specific operation of subsequent operations in the character style image generation method based on model training and text-to-image technology of the present invention

[0046] Figure 4 : The three main parts of the character style image generation method of this application.

[0047] Figure 5 : The specific process of training the target style basic model in the present invention.

[0048] Figure 6 : The specific process of character-specific LoRA model training in the present invention.

[0049] Figure 7 : The specific process of character style image reasoning and synthesis in the present invention. DETAILED DESCRIPTION

[0050] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0051] Some of the terms involved in the solution of the present invention are explained as follows:

[0052] Stable Diffusion (SD for short): is a deep learning AI model. It is a text-to-image generation technology based on the potential diffusion model. It consists of variational autoencoder VAE, U-Net and text encoder, which can generate detailed images based on text descriptions.

[0053] LoRA (Low-Rank Adaption of Large Language Models) model: is a lightweight network structure that aims to fine-tune large models through low-rank decomposition technology to adapt to specific tasks or styles. It can be understood as a plug-in of the Stable Diffusion model. Without modifying the SD model, it uses a small amount of data to train a style / IP / character to meet customized needs.

[0054] ControlNet model: is a human posture control algorithm based on deep learning, which can be understood as an "auxiliary" neural network model structure. By adding auxiliary modules to the Stable Diffusion model, it assists in controlling the generation of AI images.

[0055] Prompt: A type of instruction or information. Its core function is to provide the AI ​​model with necessary context and parameter information, guiding or triggering the AI ​​system to respond or perform corresponding operations based on these prompts.

[0056] Multi-scale Feature Fusion: It is a technology that fuses feature information of different scales. Multi-scale feature fusion technology can capture the details and features of the target object at different scales and improve the performance of target detection and recognition.

[0057] Attention mechanism: It is a technology that allows the model to focus on important information and fully learn and absorb it. This mechanism imitates the working method of human visual attention and indicates its importance by assigning weights to each element in the sequence data. In layman's terms, it focuses on important points and ignores other unimportant factors.

[0058] Blip / Blip2 algorithm (Bootstrapping Language-Image Pre-training): Blip and its advanced version Blip2 are visual language pre-training models that improve the understanding of visual scenes and the ability to generate descriptive text by combining image and text data.

[0059] Learning rate: It can be understood as the speed or intensity of learning during the LoRA model training process. Improper learning rate setting may cause overfitting or underfitting.

[0060] In one embodiment of the present invention, a method for generating a character style image based on model training and text-to-image technology is proposed. The process structure diagram is as follows: Figure 1 As shown, the whole method includes the following steps:

[0061] Train and optimize the style base model, expand the style base model training samples, improve the generalization ability of the style base model, and obtain the optimized target style basic model;

[0062] Obtain the user's character image data, pre-process the character image data, extract the user's facial feature information, proofread the facial feature information through the user feedback mechanism, and obtain the final character image training data;

[0063] Input the character image training data into the Stable Diffusion model to train the character-specific LoRA model;

[0064] Setting character style image generation information, and configuring corresponding character style image generation parameters based on the character style image generation information;

[0065] Based on the target style basic model, the character-specific LoRA model and the character style image generation parameters, text-generated images and image-generated images are processed to finally obtain the target character style image.

[0066] The character style image generation method based on model training and text-to-image technology of the present invention specifically includes:

[0067] Preliminary step: Train and optimize the style base model. For different target styles, expand the style base model training samples, improve the generalization ability of the style base model, and obtain the optimized target style basic model.

[0068] Subsequent operation 1:

[0069] S1. Obtaining the user's character image data, preprocessing the character image data, extracting the user's facial feature information based on the data, providing a user feedback mechanism, checking and correcting the facial feature information to obtain the final character image training data;

[0070] S2. Input the character image training data into the Stable Diffusion model to train the user-specific character LoRA model;

[0071] Subsequent operation 2:

[0072] S3. Setting character style image generation information, and determining corresponding generation parameters based on the character style image generation information configuration, wherein the generation information includes synthetic image style, character posture, number of generated style images, etc., and the generation parameters include style LoRA parameters, character-specific LoRA parameters, style prompt word parameters, posture control parameters, etc.;

[0073] S4. Based on the target style basic model, the character-specific LoRA model and the character style image generation parameters, text-generated image and image-generated image processing are performed to finally obtain the target character style image.

[0074] Through the above technical scheme, the present invention aims to enhance the authenticity and controllability of the synthesis results, and decomposes the character style image synthesis into three steps: target style basic model training (preliminary preparation), character-specific LoRA model training and character style image reasoning synthesis.

[0075] The target style basic model training of this application fully considers the age distortion and better synthesis effect of women than men that are common in the prior art. It collects face data samples of different genders by subdividing age groups, increases the number of samples of each age group and each gender (including but not limited to low, medium and high age groups), fully learns facial features and expressions, and improves the generalization ability of the style base model; in addition, it combines multi-scale feature fusion technology and attention mechanism to strengthen the learning of detailed features and important features of the face in the character image data, and ensures that the generated results better retain the details of the original image. The sample sources include collected user portrait photos, portrait images or dynamic videos that can be used publicly on social media or content creation platforms, documentary character clips publicly available on the Internet, etc. At the same time, the Stable Diffusion model is used to synthesize image data through prompt words and pictures.

[0076] In step S1, the user's character image data is obtained, pre-processed and facial feature information is extracted, and then the facial feature information is proofread through the user feedback mechanism, specifically including:

[0077] S11. Collecting user character image data, which is not limited to user portrait photos, but can also be multiple frames of images or dynamic videos containing user face data, such as GIF, MP4 format data, etc. Specifically, if the user provides image data, the facial feature information in the image is directly extracted after image preprocessing; if the user provides multiple frames of images or dynamic videos, the multiple frames of images or dynamic videos are first converted into an image frame set, and each frame of the image frame set is preprocessed, and then the facial feature information of the target person in the image is extracted.

[0078] S12. Preprocess the character image data. In addition to necessary operations such as non-real image filtering, facial key point detection, and image face rotation correction, image beauty style removal processing is added to reverse or reduce the impact of the beauty effect on the authenticity of the photo and restore the real characteristics of the character as much as possible.

[0079] S13. Extract facial features such as age, gender, skin condition, etc. based on the facial feature extraction algorithm, and use the user feedback mechanism to allow users to confirm whether there are errors in the facial features extracted by the algorithm, allowing users to modify and adjust, so as to further increase the accuracy of facial feature information.

[0080] In step S2, the character image training data refers to facial feature information extracted and confirmed by the user and facial image data extracted and cropped.

[0081] Furthermore, in step S3, character style image generation information is set and generation parameters are determined based thereon, specifically including:

[0082] S31. Set the target style type and the number of generated images of the character style image, and obtain the target style model id and target style prompt words (including positive prompt words and reverse prompt words) associated therewith according to the target style model id and the target style prompt words (including positive prompt words and reverse prompt words);

[0083] S32. Set the reference action of the character style image and convert it into the corresponding ControlNet character posture parameter.

[0084] Finally, the target style basic model trained and optimized in the previous step, the user-specific LoRA model trained based on the collected user character data, and the configured character style image generation parameters are combined to perform text-generated image and image-generated image processing to generate the target character style image.

[0085] The technical solution of the present invention is further described in detail below in conjunction with specific implementation methods:

[0086] It should be noted that the embodiment of the present application provides general steps for training a style-based model, and the target style can be multiple, including but not limited to ID photo style, portrait style, black and white style, children's style, Chinese style, Miao style, wedding style, etc. The base model for training the target style-based model of different styles can be consistent, and the sample parameters used for model training can be adjusted according to the corresponding style.

[0087] It should be noted that the method for generating character style images based on model training and image-text generation technology provided in the embodiment of the present application is generally executed by a GPU server.

[0088] It should be noted that Figure 2 , 3 A specific implementation method of a character style image generation method based on model training and text-to-image technology is given. It is worth noting that the method of this application is not based on Figure 2 , 3 The process sequence shown is limited.

[0089] The method of the present application mainly comprises Figure 4 The three main parts shown are:

[0090] S1. Preliminary step: target style base model training. Train and optimize the style base model. For different target styles, expand the style base model training samples, improve the generalization ability of the style base model, and obtain the optimized target style base model.

[0091] S2. Subsequent step 1: character-specific LoRA model training. Obtain the character image data of the user, pre-process the character image data, extract the user's facial feature information based on the data, establish a user feedback mechanism, check and correct the facial feature information to obtain the final character image training data, and train the character-specific LoRA model based on the character image training data;

[0092] S3. Subsequent step 2: character style image inference synthesis. Based on the character style type and posture action set by the user, determine the target style model id, target style prompt words (including positive prompt words and reverse prompt words), ControlNet parameters, posture control prompt words, etc., and based on the parameter information, combine the text-generated image and image-generated image technology to obtain the final character style image.

[0093] Figure 5 The specific process of training the target style base model is shown as follows:

[0094] S11. Select a portrait base model, such as Stable Diffusion v1.5, majicMIX_realistic_v6, ChilloutMix_Chilloutmix-Ni-pruned-fp32-fix, etc., and perform style-based LoRA model training based on the base model.

[0095] S12. The style base model training takes into full account the age distortion and better synthesis effect of women than men that are common in the prior art. It collects face data samples of different genders by age group, increases the number of samples of each age group and each gender (including but not limited to low, medium and high age groups), fully learns facial features and expressions, and improves the generalization ability of the style base model.

[0096] Specifically, the sources of training samples can be collected user portrait photos, publicly available portrait images or dynamic videos on social media or content creation platforms, documentary character clips publicly available on the Internet, etc. The Stable Diffusion model is also used to synthesize image data through prompt words, text, and images.

[0097] S13. Preprocess the collected sample data, extract each frame of the video clip, and filter out images that do not contain faces, faces that are blocked, or faces with low clarity; then perform portrait cropping on the image sample data to extract partial face data, and then use the Blip / Blip2 algorithm to encode information on each face data, that is, label each face image.

[0098] S14. Combine multi-scale feature fusion technology and attention mechanism to strengthen the learning of detailed features and important features of faces in character image data, ensuring that the generated results better retain the details of the original image;

[0099] S15. For the trained style basic model, configure corresponding style prompt word parameters, including positive prompt words and reverse prompt words, etc.

[0100] In some embodiments, the style positive prompt words can be used to guide the target style base model to generate an initial version of the image that meets the prompt word description. The positive prompt words can be used to describe the clarity of the generated image, the background color, the background complexity, the clothing of the characters in the generated image, etc. For example, for the ID photo style, the positive prompt words are high-definition ID photos, pure red background, people wearing solid color formal clothes, facing the camera. The style negative prompt words can be used to guide the target style base model to avoid generating certain specific features or elements, such as sadness, depressing atmosphere, cartoon style, low resolution, etc.

[0101] In some embodiments, the target style basic model training parameters such as network rank and network alpha, etc., appropriately increase the value of the network rank to retain more facial details. At the same time, the network alpha parameter is correspondingly set to half of the network rank or a smaller value. In addition, referring to the dynamic learning rate parameter adjustment scheme, the DAdaptAdam optimizer is selected to assist in automatically correcting the learning rate. The Adafactor optimizer can also be selected, which has lower requirements for video memory. Both can automatically correct the learning rate, which can effectively limit the occurrence of overfitting.

[0102] Figure 6 The specific implementation of S2 subsequent step 1-character-specific LoRA model training is shown as follows:

[0103] S21. Obtain the user's character image data to prepare for subsequent user-specific LoRA model training.

[0104] In this embodiment, the user character image data can be a personal picture uploaded by the user, or a multi-frame image or a dynamic video containing the user's facial information, that is, data in multiple formats such as PNG, JPG, JPEG, PNG, and MP4 are supported. For the user's personal picture or the dynamic video containing the user's facial information, it can be an image or dynamic video clip stored locally by the user, or it can be an image or dynamic video clip that the user shoots online and then uploads. Therefore, in this embodiment, there are multiple ways to collect user character image data, so that the user can choose according to the existing available resources or specific needs to adapt to the actual conditions of different users.

[0105] In this embodiment, the user character image data in non-picture format is converted into a picture frame set for subsequent image processing;

[0106] S22. Preprocess the user's character image data to ensure that the trained LoRA model can learn more detailed character image features. The specific process is as follows:

[0107] S221. Remove the beauty effect in the picture by using algorithms or image editing software;

[0108] S222. Use the face detection algorithm and the face key point detection algorithm to extract the face in the image and locate the face area, filter out the data that does not contain faces, is not a real face, and has occluded faces; calculate the area ratio of the face in the image, and filter out the data with a face ratio that is too small. For images containing multiple faces, use the user feedback mechanism to let the user confirm the final character image that needs to be retained;

[0109] S223. Using the image face rotation algorithm, the face data obtained in step 2 is rotated and aligned, that is, after processing, the line connecting the center coordinates of the left and right eyes of the face is in a horizontal state;

[0110] S224: Encode information on the face image processed by steps 1-3. First, encode information on the face image using an image-based text encoder and decoder. Then, the encoded information data is confirmed by the user through a user feedback mechanism. Finally, the encoded information data and the face image are input into a portrait base model to train a user-specific LoRA model.

[0111] In this embodiment, an algorithm or image editing software is used to remove the beautification effect in a picture. First, the face data of the image to be processed is input into the algorithm or image editing software to detect whether the image data contains the beautification effect. For image data containing the beautification effect, key features such as texture and contour are extracted from the image. Then, according to the type of beautification effect detected, a suitable inverse processing algorithm or tool is selected to process the image to gradually remove the beautification effect.

[0112] Through the preprocessing operation of this embodiment, image data that does not contain faces, is not a real face, has occluded faces, and has a face ratio that is too small can be filtered out. In addition, for a data set containing multiple face images, all the face data that appear is first counted, and then sorted in descending order according to the frequency of occurrence, and the face with the highest frequency is selected for user confirmation through the user feedback mechanism. This method avoids the complexity of providing multiple options for users to choose from, simplifies the user confirmation process, and ensures the accuracy of the training data and user satisfaction.

[0113] Since the face angle and the forward angle in the collected character image data often have a large deviation, if it is directly used for model training without processing, it will seriously affect the output result of the model. In this embodiment, the face image data that meets the requirements is subjected to image face rotation alignment processing, so that the line connecting the center coordinates of the left and right eyes of the face in the face data used for user-specific LoRA model training is in a horizontal state. The image face rotation alignment model can refer to the Rot-bgr model, the Dlib68 model, etc.

[0114] In some embodiments, a face detection algorithm and a face key point detection algorithm are used to extract faces from images and locate face areas. The face detection algorithm and the face key point detection algorithm can refer to the Retina-Face face detection key point model, the MTCNN face detection key point model, the Huawei CenterFace model, the Google BlazeFace model, and the like.

[0115] In some embodiments, a variety of image processing software and algorithms, such as Adobe Photoshop, Primo, ImgMeta, etc., can be used, and deep learning algorithms, such as convolutional neural networks (CNN), etc., can also be used to achieve automated beauty effect detection and reversal.

[0116] In some embodiments, the pre-processed image face data is first encoded with the help of algorithms such as Blip / Blip2. Specifically, the information encoding includes the person's gender, hair color, skin color, whether glasses are worn, glasses color, facial decorations (such as earrings, nose rings, etc.). Then, the person's gender and age are extracted with the help of facial attribute recognition algorithms such as FairFace and Baidu facial attribute recognition interface. Finally, the information encoding obtained by the algorithm is fed back to the user for confirmation. At the same time, the user is also supported to modify and add or subtract his or her own facial features, such as the shape of facial features, age, etc., to obtain the final version of the user's facial information encoding data.

[0117] In some embodiments, according to preset training parameters, such as training step size, rounds, epochs, etc., the user's facial image data and corresponding information encoding data are input together for model training, and finally a user-specific LoRA model is obtained.

[0118] Figure 7 The specific implementation of S3 subsequent step 2 - character style image inference synthesis is shown, as detailed below:

[0119] S31. Obtain the target style and the number of synthetic character style images selected by the user;

[0120] S32. Based on the target style selected by the user, determine the target style synthesis parameters, including the target style basic model, target style prompt word parameters (including positive prompt words and reverse prompt words);

[0121] S33. Obtain the posture reference image selected by the user, determine the posture skeleton image based on the posture reference image, and finally determine the target style ControlNet posture control parameters;

[0122] S34. Based on the target style synthesis parameters, the posture control parameters, and the user-specific LoRA model, the Wensheng image is processed to generate an initial version of the character style image set, and then the face area in the initial version of the character style image set and any face data used for user-specific LoRA model training are subjected to image-face nesting fusion processing to obtain an enhanced version of the character style image set;

[0123] S35. Calculate the similarity between the face part of the image in the enhanced version character style image set and the face part of any character image data uploaded by the user that meets the specification, and sort the similarity results in descending order, and output the top n character style images with the highest similarity scores to the user, i.e., the final character style images, where n is the number of synthesized character style images selected by the user.

[0124] In some embodiments, the posture of the character in the generated image is controlled using the ControlNet posture control parameters. Specifically, the posture control parameters may be general parameters pre-configured according to the target style basic model, or may be posture control parameters converted from a posture reference image selected by the user. The specific conversion method may refer to the records in the relevant technology, which will not be described here.

[0125] In some embodiments, the number of the initial version task style image set can be a specified multiple (greater than 1) of the preset maximum number of synthesized images, such as 1.5 times, or a specified multiple (greater than 1) of the number of synthesized character style images selected by the user. Performing image-face nesting fusion processing on the face area in the initial version character style image set and the face data used for user-specific LoRA model training can improve the consistency between the face of the synthesized character style image and the actual face of the user.

[0126] In some embodiments, a similarity is calculated between the face area in the enhanced version of the character style image set and the face area of ​​any character image data uploaded by the user that meets the specifications, and the top n results are output to the user in descending order of similarity value. This method reduces the workload of the user in screening the character style image synthesis results on the one hand, and improves the accuracy and authenticity of the output character style image results on the other hand.

[0127] It should be noted that by synthesizing character style images in this embodiment, the privacy of users is strictly protected. The character image data uploaded by users and the user-specific LoRA model trained are encrypted and stored on the internal server. The synthesized character style images will also be marked with digital watermarks to reduce the risk of AI "deep fakes".

[0128] In another embodiment, the present invention proposes a character style image generation system based on model training and image-text generation technology, including:

[0129] The target style base model optimization module is used to train and optimize the style base model, expand the style base model training samples, improve the generalization ability of the style base model, and obtain the optimized target style base model;

[0130] The character image training data acquisition module is used to obtain the character image data of the user, pre-process the character image data, extract the user's facial feature information, proofread the facial feature information through the user feedback mechanism, and obtain the final character image training data;

[0131] The LoRA model training module is used to input character image training data into the Stable Diffusion model to train a character-specific LoRA model;

[0132] A parameter configuration module, used to set character style image generation information and configure corresponding character style image generation parameters based on the character style image generation information;

[0133] The character style image generation module is used to perform text-generated image and image-generated image processing based on the target style basic model, the character-specific LoRA model and the character style image generation parameters, and finally obtain the target character style image.

[0134] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for generating a character style image based on model training and text-to-image technology as described above is implemented.

[0135] In another embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program, wherein the computer program enables a computer to execute the method for generating a character style image based on model training and text-to-image technology as described above.

[0136] The application scenarios of the technical solution of the present invention include:

[0137] Personalized entertainment scenes allow users to obtain personal photos of various styles without leaving home, satisfying their entertainment needs;

[0138] ID photo synthesis, no need to go to a photo studio, no need to change clothes / makeup, just upload a few personal life photos or a short dynamic video, you can synthesize multiple standard ID photos, and can ensure that the synthesis result and real face recognition pass rate exceeds 95%. For example, ID card photo synthesis, passport photo synthesis, etc., help reduce shooting costs;

[0139] E-commerce product scene image synthesis. Using this technical solution can improve the efficiency of e-commerce promotional poster production, reduce high site layout fees, shooting fees, etc., and save costs.

[0140] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0141] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for generating a character style image based on model training and text-to-image technology, characterized in that: The following steps are involved: Train and optimize the style base model, expand the style base model training samples, improve the generalization ability of the style base model, and obtain the optimized target style basic model; Obtain the user's character image data, pre-process the character image data, extract the user's facial feature information, proofread the facial feature information through the user feedback mechanism, and obtain the final character image training data; Input the character image training data into the Stable Diffusion model to train the character-specific LoRA model; Setting character style image generation information, and configuring corresponding character style image generation parameters based on the character style image generation information; Based on the target style basic model, the character-specific LoRA model and the character style image generation parameters, text-generated images and image-generated images are processed to finally obtain the target character style image.

2. The method for generating a character style image based on model training and text-to-image technology according to claim 1 is characterized in that: The obtaining of the user's character image data is specifically as follows: using the AIGC model tool to generate different character image data, the character images of the character image data cover samples of all age groups and genders, fully learning facial features and expressions, combining multi-scale feature fusion technology and attention mechanism, and strengthening the learning of detailed features and important features of the face in the character image data.

3. The method for generating a character style image based on model training and text-to-image technology according to claim 1, characterized in that: The pre-processing of the character image data includes non-real image filtering, facial key point detection, image face rotation correction and image beauty style removal processing; The facial feature information of the user is extracted and proofread through a user feedback mechanism, specifically: facial feature information is extracted based on a facial feature extraction algorithm, and through a user feedback mechanism, the user is allowed to confirm whether there are errors in the facial feature information extracted by the algorithm, and the user is allowed to modify and adjust.

4. The method for generating a character style image based on model training and text-to-image technology according to claim 3 is characterized in that: The specific process of preprocessing the character image data is as follows: (1) Use algorithms or image editing software to remove the beauty effects in the image; (2) Use face detection algorithms and face key point detection algorithms to extract faces from images and locate face areas, filter out data that does not contain faces, is not a real face, or has faces that are obscured; calculate the area ratio of faces in the image and filter out data where the face ratio is too small; for images containing multiple faces, use a user feedback mechanism to let users confirm the characters that need to be retained; (3) using an image face rotation algorithm, the face data obtained in step (2) is rotated and aligned so that the line connecting the center coordinates of the left and right eyes of the face is in a horizontal state after processing; (4) Encoding information on the facial image processed by steps (1) to (3): first, encoding information on the facial image using an image-based text encoder and decoder; then, allowing the user to confirm the information encoded data through a user feedback mechanism; and finally, inputting the information encoded data and the facial image into a portrait base model to train a user-specific LoRA model.

5. The method for generating a character style image based on model training and text-to-image technology according to claim 1 is characterized in that: If the user provides image data, the facial feature information in the image is directly extracted after image preprocessing; if the user provides multiple-frame images or dynamic videos, the multiple-frame images or dynamic videos are first converted into an image frame set, each frame in the image frame set is preprocessed, and then the facial feature information of the target person in the image is extracted.

6. The method for generating a character style image based on model training and text-to-image technology according to claim 1, characterized in that: The step of setting the character style image generation information and configuring corresponding character style image generation parameters based on the character style image generation information includes: Set the target style type and number of generated images of the character style image, and obtain the associated target style model id and target style prompt word based on them; Set the reference action of the character style image and convert it into the corresponding ControlNet character pose parameters.

7. The method for generating a character style image based on model training and text-to-image technology according to claim 1, characterized in that: The target style basic model, the character-specific LoRA model and the character style image generation parameters are used to perform text-generated image and image-generated image processing to finally obtain the target character style image. The specific process is as follows: (1) Obtain the target style and the number of synthesized character style images selected by the user; (2) determining target style synthesis parameters based on the target style selected by the user, including a target style basic model and target style prompt word parameters; (3) Obtain the posture reference image selected by the user, determine the posture skeleton image based on the posture reference image, and finally determine the target style ControlNet posture control parameters; (4) performing image processing based on the target style synthesis parameters, the posture control parameters, and the user-specific LoRA model to generate an initial version of the character style image set, and then performing image-face nesting fusion processing on the face area in the initial version of the character style image set and any face data used for training the user-specific LoRA model to obtain an enhanced version of the character style image set; (5) Calculate the similarity between the facial part of the image in the enhanced version of the character style image set and the facial part of any character image data uploaded by the user that meets the specifications, and sort the similarity results in order from large to small, and output the top n character style images with the highest similarity scores to the user, that is, the final character style image; where n is the number of synthesized character style images selected by the user.

8. A character style image generation system based on model training and text-to-image technology, characterized in that: include: The target style base model optimization module is used to train and optimize the style base model, expand the style base model training samples, improve the generalization ability of the style base model, and obtain the optimized target style base model; The character image training data acquisition module is used to obtain the character image data of the user, pre-process the character image data, extract the user's facial feature information, proofread the facial feature information through the user feedback mechanism, and obtain the final character image training data; The LoRA model training module is used to input character image training data into the Stable Diffusion model to train a character-specific LoRA model; A parameter configuration module, used to set character style image generation information and configure corresponding character style image generation parameters based on the character style image generation information; A character style image generation module is used to perform text-generated image and image-generated image processing based on the target style basic model, the character-specific LoRA model and the character style image generation parameters, and finally obtain the target character style image; The system is used to execute the character style image generation method based on model training and image-text generation technology as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for generating a character style image based on model training and text-to-image technology as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for generating a character style image based on model training and image-text generation technology as described in any one of claims 1 to 7.

Images