Image processing method, apparatus, readable storage medium and program product
By performing feature removal and stitching on user face images, combined with fine-tuning of the LoRa model, the problem of poor image local feature replacement effect in existing technologies has been solved, achieving efficient image editing effect with a small number of samples.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAMEN MEITUZHIJIA TECH
- Filing Date
- 2026-04-01
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies require a large number of perfectly matched training samples for image local editing tasks, especially when replacing features such as bangs, eyebrows, eyes, or mouth. However, these samples are difficult to obtain in reality, causing the model to be unable to edit accurately with limited data.
By performing feature removal processing on the first face images of multiple users, various second feature images are generated. The initial image editing model is then fine-tuned using the LoRa model and stitched together with image masks to form the initial image editing model for local editing.
Even with a small number of training samples, it can effectively improve the effect of image replacement editing and achieve accurate generation of local image features.
Smart Images
Figure CN122492865A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, readable storage medium, and program product. Background Technology
[0002] For specific image local editing tasks, such as wanting to change at least one of the following in a user image: bangs type, eyebrow shape, eye shape, or mouth shape, existing methods, such as InstructPix2Pix and MagicBrush, generally require hundreds of thousands of paired data as training samples. However, such perfectly paired training samples are difficult to obtain in the real world. Therefore, with a limited amount of training sample data, the trained model cannot accurately perform replacement-style editing on the user image, which severely restricts the image editing effect. Summary of the Invention
[0003] Based on this, this application provides an image processing method, apparatus, readable storage medium, and program product, which can effectively improve the image replacement editing effect.
[0004] On the one hand, this application provides an image processing method, including:
[0005] The first facial images of multiple users are processed by removing the first feature to obtain the processed image;
[0006] Based on the processed image, a second face image of each user is generated, including multiple second features; the first feature and the second feature are different morphological features of the same location or hair type.
[0007] For each user, the processed image of the user, the second face image, and the corresponding image mask are stitched together to obtain multiple first stitched images of the user;
[0008] Based on the first stitched image of each user and the first prompt word for the generation of body parts or hair, the Lora model in the initial image editing model is fine-tuned, and the resulting first image editing model is deployed; wherein, the initial image editing model is obtained by embedding the Lora model in a pre-trained Flux-fill model, and the Lora model is used to extract body part features or hair features.
[0009] Upon receiving a first user request, the first image editing model is deployed to perform local editing on the user image to be processed based on the second prompt word in the first user request, thereby obtaining the face image corresponding to the second prompt word.
[0010] In one embodiment, the first feature and the second feature are different eyebrow shapes; or,
[0011] The first feature and the second feature are different bangs hairstyles; or,
[0012] The first feature and the second feature are different eye shapes; or,
[0013] The first feature and the second feature are different mouth shapes.
[0014] In one embodiment, the first feature and the second feature are different bangs hairstyles, the first face image is a first bangs image, the processed image is a no-bangs image, the second face image is a second bangs image, and the Lora model is used to extract bangs hairstyle features.
[0015] The first feature removal process for the first face images of multiple users to obtain the processed image includes: deleting the bangs hairstyle in the first bangs images of multiple users to obtain an image without bangs;
[0016] The process of generating second face images of each user, including multiple second features, based on the second face image includes: generating second bangs images of each user with multiple bangs hairstyles based on the image without bangs;
[0017] The step of stitching together the processed image, the second face image, and the corresponding image mask of each user to obtain multiple first stitched images of the user includes: stitching together the user's image without bangs and the image mask without bangs with the second bangs image and the second bangs image mask of each bangs hairstyle to obtain multiple first stitched images of the user with and without bangs.
[0018] The step of fine-tuning the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word for generating the bangs hairstyle includes: fine-tuning the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word for generating the bangs hairstyle;
[0019] The step of performing local editing processing on the user image to be processed based on the second prompt word in the first user request to obtain the face image corresponding to the second prompt word includes: using the deployed first image editing model to perform bangs hairstyle generation processing on the second prompt word in the first user request and the user image to be processed to obtain the bangs image corresponding to the second prompt word.
[0020] In one embodiment, the step of deleting the bangs hairstyle from the first bangs images of multiple users to obtain bangs-free images includes:
[0021] Call the first model interface of the notch removal model;
[0022] The first model interface is used to input the first bangs images of each user in batches, so that the bangs removal model can delete the bangs hairstyle in the first bangs images of each user and output a bangs-free image.
[0023] In one embodiment, generating a second bangs image based on the bangs-free image for each user with multiple bangs hairstyles includes:
[0024] Obtain third-party suggestions for generating various bangs hairstyles;
[0025] Call the second model interface of the bangs generation model;
[0026] By batch inputting the no-bangs images of each user and the corresponding third prompt words through the second model interface, the bangs generation model can generate second bangs images of various bangs hairstyles for each user based on the no-bangs images and the third prompt words.
[0027] In one embodiment, fine-tuning the Lora model in the initial image editing model based on the stitched images of each user and the first prompt word for generating the bangs hairstyle includes:
[0028] For each user, the stitched image of the user and the first prompt word for generating bangs hairstyle are input into the initial image editing model, so that the initial image editing model performs bangs hairstyle generation processing based on the first prompt word and the stitched image to obtain predicted bangs images of various bangs hairstyles for the user;
[0029] The loss value is determined based on the predicted bangs image and the second bangs image in the stitched image;
[0030] The Lora model in the initial image editing model is fine-tuned based on the loss value so that the Lora model learns the bangs hairstyle features of each of the bangs hairstyles.
[0031] In one embodiment, the step of generating a bangs hairstyle image by using the deployed first image editing model to process the second prompt word in the first user request and the user image to be processed, thereby obtaining the bangs image corresponding to the second prompt word, includes:
[0032] Extract the second prompt word and the user image to be processed from the first user request;
[0033] The first image editing model is invoked based on the first user request;
[0034] The second prompt word and the user image are input into the first image editing model, so that the first image editing model uses the fine-tuned parameters of the Lora model and the parameters of the Flux-fill model to generate a bangs hairstyle for the user image to be processed based on the second prompt word, thereby obtaining the bangs image corresponding to the second prompt word.
[0035] On the one hand, this application also provides an image processing apparatus, including:
[0036] The deletion module is used to remove the first feature from the first face images of multiple users to obtain the processed image;
[0037] The first generation module is used to generate a second face image for each user, including multiple second features, based on the processed image; the first feature and the second feature are different morphological features of the same location or hair type.
[0038] The stitching module is used to stitch together the processed image, the second face image, and the corresponding image mask of each user to obtain multiple first stitched images of the user.
[0039] The model adjustment module is used to fine-tune the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word for the generation of body parts or hair, and to deploy the resulting first image editing model; wherein, the initial image editing model is obtained by embedding the Lora model in a pre-trained Flux-fill model, and the Lora model is used to extract body part features or hair features.
[0040] The second generation module is used to, upon receiving a first user request, perform local editing processing on the user image to be processed based on the second prompt word in the first user request using the deployed first image editing model, to obtain the face image corresponding to the second prompt word.
[0041] In one embodiment, the first feature and the second feature are different eyebrow shapes; or, the first feature and the second feature are different bangs hairstyles; or, the first feature and the second feature are different eye shapes; or, the first feature and the second feature are different mouth shapes.
[0042] In one embodiment, the first feature and the second feature are different bangs hairstyles, the first face image is a first bangs image, the processed image is a no-bangs image, the second face image is a second bangs image, and the Lora model is used to extract bangs hairstyle features.
[0043] The deletion module is also used to delete the bangs hairstyle in the first bangs images of multiple users to obtain bangs-free images;
[0044] The first generation module is further configured to generate second bangs images of various bangs hairstyles for each user based on the bangs-free image;
[0045] The splicing module is also used to splice the user's image without bangs and the image mask without bangs with the second image and the second image mask of each of the bangs hairstyles, respectively, to obtain multiple first spliced images of the user with and without bangs.
[0046] The model adjustment module is also used to fine-tune the Lora model in the initial image editing model based on the first spliced image of each user and the first prompt word used to generate the bangs hairstyle;
[0047] The second generation module is further configured to use the deployed first image editing model to generate a bangs hairstyle from the second prompt word in the first user request and the user image to be processed, thereby obtaining a bangs image corresponding to the second prompt word.
[0048] In one embodiment, the deletion module is further configured to call the first model interface of the bangs removal model; and to input the first bangs images of each user in batches through the first model interface, so that the bangs removal model deletes the bangs hairstyle in the first bangs images of each user and outputs a bangs-free image.
[0049] In one embodiment, the first generation module is further configured to obtain a third prompt word for generating various bangs hairstyles; call the second model interface of the bangs generation model; and input in batches the bangs-free images and the corresponding third prompt words of each user through the second model interface, so that the bangs generation model generates second bangs images of various bangs hairstyles of each user based on the bangs-free images and the third prompt words.
[0050] In one embodiment, the model adjustment module is further configured to, for each user, input the stitched image of the user and a first prompt word for generating bangs hairstyles into an initial image editing model, so that the initial image editing model performs bangs hairstyle generation processing based on the first prompt word and the stitched image to obtain predicted bangs images of various bangs hairstyles for the user; determine a loss value based on the predicted bangs image and a second bangs image in the stitched image; and fine-tune the Lora model in the initial image editing model based on the loss value, so that the Lora model learns the bangs hairstyle features of each bangs hairstyle.
[0051] In one embodiment, the second generation module is further configured to extract a second prompt word and a user image to be processed from the first user request; invoke the deployed first image editing model based on the first user request; input the second prompt word and the user image into the first image editing model, so that the first image editing model uses the fine-tuned parameters of the Lora model and the parameters of the Flux-fill model to generate a bangs hairstyle for the user image to be processed based on the second prompt word, thereby obtaining a bangs image corresponding to the second prompt word.
[0052] On the one hand, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image processing method.
[0053] In one aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image processing method.
[0054] On the one hand, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image processing method.
[0055] The aforementioned image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product perform first feature removal processing on first face images of multiple users to obtain processed images; based on the processed images, generate second face images for each user including multiple second features, where the first and second features are different morphological features of the same body part or hair type, thereby obtaining a large number of face images with and without the first and second features; furthermore, for each user, stitch together the user's processed image, second face image, and corresponding image mask to obtain multiple first stitched images for the user; based on the first stitched images of each user and the first prompt words used for body part or hair type generation, fine-tune the Lora model in the initial image editing model, and deploy the resulting first image editing model; wherein, the initial image editing model is based on the pre-trained Flux-f... The image editing model is obtained by embedding a Lora model into the IL model. The Lora model is used to extract part features or hair features. Therefore, even if there are only a few face images with first and second features, the initial image editing model can be trained, enabling the model to learn to generate images with first or second features from images without first and second features. In addition, considering that the Lora model is used to extract part features or hair features, after fine-tuning the Lora model, it can effectively improve the replacement editing effect of the image editing model compared to the conventional Flux-fill model. Therefore, when a first user request is received, the deployed first image editing model performs local editing processing on the user image to be processed based on the second prompt word in the first user request to obtain the face image corresponding to the second prompt word, thereby effectively improving the replacement editing effect of the image. Attached Figure Description
[0056] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0057] Figure 1 This is an application environment diagram of an image processing method in one embodiment;
[0058] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;
[0059] Figure 3 This is a schematic diagram of the model structure for the training and inference phases in one embodiment;
[0060] Figure 4This is a structural block diagram of an image processing device in one embodiment;
[0061] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0062] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0063] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.
[0064] Taking the editing of hairstyles in a face image as an example, server 104 can delete the bangs hairstyle in the first bangs images of multiple users to obtain bangs-free images; based on the bangs-free images, it generates second bangs images of various bangs hairstyles for each user; for each user, the user's bangs-free image and its mask are respectively stitched with the second bangs images and their masks for each bangs hairstyle to obtain multiple first stitched images of the user with and without bangs; based on the first stitched images of each user and the first prompt word used to generate bangs hairstyles, the Lora model in the initial image editing model is fine-tuned, and the resulting first images are then processed. The image editing model is deployed; the initial image editing model is obtained by embedding a Lora model into a pre-trained Flux-fill model. The Lora model is used to extract the features of the bangs hairstyle. This image editing model can be deployed on server 104 or on terminal 102. If deployed on server 104, terminal 102 can send a first user request for image processing to server 104 when it needs to generate a bangs hairstyle. The image editing model of server 104 performs bangs hairstyle generation processing on the second prompt word in the first user request and the user image to be processed to obtain the bangs image corresponding to the second prompt word.
[0065] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0066] In one exemplary embodiment, such as Figure 2 As shown, an image processing method is provided, which is applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 202 to 210. Wherein:
[0067] Step 202: Perform first feature removal processing on the first face images of multiple users to obtain the processed image.
[0068] Among them, a face image refers to a person image containing a face, such as a head face image, an upper body image, or a full-body image.
[0069] The primary feature can be a morphological characteristic of a certain part or hair type. The part can be the mouth, eyes, or other facial features; the hair type can be hair, eyebrows, or beard. For example, if the hair is hair, the primary feature can be the bangs or hairstyle of the hair; if the hair is eyebrows or beard, the primary feature can be the shape of the eyebrows or the shape of the beard.
[0070] In one embodiment, for a scenario where bangs need to be generated or replaced, the first face image is the first bangs image, and the server can delete the bangs hairstyles in the first bangs images of multiple users to obtain images without bangs.
[0071] The first bangs image can be the user's original bangs hairstyle image (referred to as the bangs image). For the same user, there can be one or more bangs hairstyle images. The image without bangs can be the image after removing the bangs hairstyle features from the first bangs image, that is, the image after removing the bangs.
[0072] Bangs hairstyles, also known as bangs styles, include blunt bangs, French bangs, side-cut bangs, Japanese wet bangs, and airy bangs, etc. Correspondingly, a user's first bangs image can be at least one of the following: blunt bangs, French bangs, side-cut bangs, Japanese wet bangs, or airy bangs.
[0073] In one embodiment, the server can call the first model interface of the bangs removal model; and input the first bangs images of each user in batches through the first model interface, so that the bangs removal model can delete the bangs hairstyle in the first bangs images of each user and output an image without bangs.
[0074] The bangs removal model can be a self-developed AI model for removing bangs, achieving a relatively realistic bangs removal effect. For example, the server can call the company's model interface, input a batch of collected images of thin bangs, and output images without bangs after removal.
[0075] In another embodiment, for scenarios where modifications to the shape of eyebrows, eyes, mouth, or beard are required, the server can delete the shape of eyebrows, eyes, mouth, or beard from the first face images of multiple users to obtain an image without eyebrows, eyes, mouth, or beard.
[0076] The mustache shape can be a mustache with two strokes, or other shapes. The eyebrow shape can be a drooping eyebrow, a standard eyebrow, a thick, flat eyebrow, a willow leaf eyebrow, a European-style eyebrow, or a sword eyebrow, etc. The eye shape can be an almond eye, a phoenix eye, or other shapes. The mouth shape can be a thin lip, a thick lip, an arched lip, an M-shaped lip, or a cherry lip, etc.
[0077] In one embodiment, the server can call the first model interface of other part or hair removal models (hereinafter referred to as target removal models); and input the first face images of each user in batches through the first model interface, so that the target removal model can delete the eyebrows, eyes, mouth or beard in the first face images of each user, and output the processed image without eyebrows, eyes, mouth or beard.
[0078] Among them, the target removal model is similar to the bangs removal model, which can be an artificial intelligence model developed by the company to eliminate eyebrow shape, eye shape, mouth shape or mustache shape.
[0079] Step 204: Generate a second face image for each user based on the processed image, including multiple second features; the first feature and the second feature are different morphological features under the same location or hair type.
[0080] Among them, the first feature and the second feature are different eyebrow shapes; or, the first feature and the second feature are different bangs hairstyles; or, the first feature and the second feature are different eye shapes; or, the first feature and the second feature are different mouth shapes.
[0081] Taking hair as an example, if the first feature is a blunt, thin fringe hairstyle, then multiple second features can be French fringe, slanted fringe, Japanese wet fringe, and airy fringe, etc. In some practical applications, multiple second features may include not only the aforementioned fringe hairstyles, but also blunt, thin fringe hairstyles.
[0082] In one embodiment, for scenarios where bangs need to be generated or replaced, the second face image is a second bangs image, and the server can generate second bangs images of various bangs hairstyles for each user based on the bangs-free image.
[0083] The second bangs image can be a bangs image generated from a bangs-free image, showing a bangs hairstyle.
[0084] In one embodiment, the server can obtain a third prompt word for generating various bangs hairstyles; call the second model interface of the bangs generation model; and input each user's bangs-free image and corresponding third prompt word in batches through the second model interface, so that the bangs generation model can generate second bangs images of various bangs hairstyles for each user based on the bangs-free image and the third prompt word.
[0085] The base model used by the notch generation model can be the same as that used by the notch removal model, such as using an image editing model without a LoRa model. However, the prompt text used during training will be different. In some practical applications, the functions of the notch generation model and the notch removal model can be integrated into the same large language model. For example, given different prompt words and corresponding face images, the large language model can generate or remove notches from the face image based on the prompt words.
[0086] In another embodiment, for scenarios where modifications to the eyebrow shape, eye shape, mouth shape, or mustache shape are required, the server can generate a second face image for each user with multiple eyebrow shapes, eye shapes, mouth shapes, or mustache shapes based on the processed image.
[0087] Specifically, the server can obtain prompts for generating various eyebrow, eye, mouth, or mustache shapes; call the model interface of other body part or hair generation models (hereinafter referred to as target generation models); and input the processed images and corresponding prompts of each user in batches through the model interface, so that the target generation model can generate second face images of each user with various eyebrow, eye, mouth, or mustache shapes based on the image without bangs and the third prompt.
[0088] The base model used by the target generation model can be the same as that used by the target removal model, such as using an image editing model without a LoRa model. However, the prompt text used during training is different.
[0089] Step 206: For each user, the processed image, the second face image, and the corresponding image mask of the user are stitched together to obtain multiple first stitched images of the user.
[0090] The image mask can include the image mask corresponding to the processed image and the image mask corresponding to the second face image. For example, if the processed image is a bangs-free image and the second face image is a bangs-free image, the image mask can include a bangs-free image mask and a bangs-free image mask. The bangs-free image mask can be an image mask with a pixel value of 0, and the bangs-free image mask can be an image mask with a pixel value of 1.
[0091] Image masks can be drawn by writing code, and their height and width can be 512 and 1024, respectively.
[0092] Therefore, the first stitched image can be a four-grid image, including the processed image, the image mask corresponding to the processed image, the second face image, and the image mask corresponding to the second face image.
[0093] For example, for each user, the user's processed image, the image mask corresponding to the processed image, the second face image, and the image mask corresponding to the second face image are stitched together to obtain the user's first stitched image. For example, a 512*512 image without bangs and a 512*512 image with bangs are stitched together to form a 512*1024 paired image a. Then, two image masks of size 512*512 (such as two image masks with black on the left and white on the right) are stitched together to form mask b. Finally, a four-grid stitched image of paired image a and mask b is obtained.
[0094] In one embodiment, for scenarios where bangs need to be generated or replaced, the second face image is a second bangs image for various bangs hairstyles. The server can, for each user, stitch the user's no-bangs image and no-bangs image mask with the second bangs image and second bangs image mask for each bangs hairstyle, respectively, to obtain multiple first stitched images of the user with and without bangs.
[0095] In another embodiment, for scenarios where modifications to eyebrow shape, eye shape, mouth shape, or mustache shape are required, the server can stitch together the user's processed image, the second face image of each eyebrow shape, eye shape, mouth shape, or mustache shape, and the corresponding image mask for each user to obtain multiple first stitched images of the user with or without eyebrow shape, eye shape, mouth shape, or mustache shape.
[0096] Step 208: Based on the first stitched image of each user and the first prompt word for the location or hair generation, fine-tune the Lora model in the initial image editing model, and deploy the resulting first image editing model.
[0097] The initial image editing model is obtained by embedding a Lora model into a pre-trained Flux-fill model. The Lora model is used to extract part features or hair features.
[0098] In one embodiment, for a scenario where bangs need to be generated or replaced, the server can fine-tune the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word used to generate the bangs hairstyle, and then deploy the resulting first image editing model; wherein, the Lora model is used to extract the features of the bangs hairstyle.
[0099] In one embodiment, the server can input a user's stitched image and a first prompt word for generating bangs hairstyles into an initial image editing model for each user. The initial image editing model then generates bangs hairstyles based on the first prompt word and the stitched image, resulting in predicted bangs images for various bangs hairstyles for the user. A loss value is determined based on the predicted bangs images and a second bangs image from the stitched image. The Lora model in the initial image editing model is then fine-tuned based on the loss value, so that the Lora model learns the bangs hairstyle features for each bangs hairstyle.
[0100] In one embodiment, the server can also adjust the expressions of second bangs images of various bangs hairstyles to obtain third bangs images of various bangs hairstyles; wherein the expressions of the third bangs images of various bangs hairstyles differ; for each user, the user's image without bangs and the image mask without bangs are respectively stitched with the third bangs images and the third bangs image masks of each bangs hairstyle to obtain multiple second stitched images of the user with and without bangs; based on the second stitched images of each user and the fourth prompt word used to generate the bangs hairstyle, the parameters of the initial image editing model are adjusted, and the obtained second image editing model is deployed; upon receiving a second user request, the deployed second image editing model is used to generate bangs hairstyles based on the fourth prompt word in the second user request and the user image to be processed, to obtain the bangs image corresponding to the fourth prompt word; wherein the expression in the bangs image corresponding to the fourth prompt word is different from the expression in the user image.
[0101] In one embodiment, the server can extract key facial features of a user from second images of various bangs hairstyles; obtain expression models that match various bangs hairstyles; and adjust the user's expression in each second bangs image based on the key facial features and expression models to obtain a third bangs image of various bangs hairstyles.
[0102] In addition, for scenarios that require modification of eyebrow shape, eye shape, mouth shape, or mustache shape, the initial image editing model can be trained using the training scheme for the above-mentioned bangs generation or replacement scenario.
[0103] Step 210: Upon receiving a first user request, the first image editing model is deployed to perform local editing processing on the user image to be processed based on the second prompt word in the first user request, thereby obtaining the face image corresponding to the second prompt word.
[0104] In one embodiment, for a scenario where bangs need to be generated or changed, the server can, upon receiving a first user request, use a deployed first image editing model to generate a bangs hairstyle based on the second prompt word in the first user request and the user image to be processed, thereby obtaining the bangs image corresponding to the second prompt word.
[0105] Specifically, the server can extract the second prompt word and the user image to be processed from the first user request; call the deployed first image editing model based on the first user request; input the second prompt word for adjusting the bangs hairstyle and the corresponding user image into the first image editing model, so that the first image editing model uses the fine-tuned parameters of the Lora model and the parameters of the Flux-fill model to generate the bangs hairstyle based on the second prompt word in the user image to be processed (such as editing the hairstyle in the user image to be processed according to the second prompt word), and obtain the bangs image corresponding to the second prompt word.
[0106] In another embodiment, if the first image editing model is trained using an eyebrow image, an eye image, a mouth image, or a mustache image and corresponding prompt words, the server can also receive a first user request for modification of the eyebrow, eye, mouth, or mustache shape. By deploying the first image editing model, the server modifies the eyebrow, eye, mouth, or mustache shape in the user image to be processed according to the second prompt word in the first user request, and obtains the bangs image corresponding to the second prompt word.
[0107] In the above embodiments, a first feature removal process is performed on the first face images of multiple users to obtain processed images; based on the processed images, a second face image of each user is generated, including multiple second features. The first and second features are different morphological features under the same location or hair type, thereby obtaining a large number of face images with and without the first and second features; furthermore, for each user, the processed image, the second face image, and the corresponding image mask are stitched together to obtain multiple first stitched images of the user; based on the first stitched images of each user and the first prompt words used for location or hair generation, the Lora model in the initial image editing model is fine-tuned, and the resulting first image editing model is deployed; wherein, the initial image editing model is a pre-trained Flux-fill model with an embedded Lora model. Therefore, the LoRa model is used to extract particulate or hair features. Thus, even with only a small number of face images containing only the first and second features, the initial image editing model can be trained, enabling the model to learn to generate images with either the first or second feature from images lacking them. Furthermore, considering that the LoRa model is used to extract particulate or hair features, fine-tuning the LoRa model can effectively improve the replacement-style editing effect compared to the conventional Flux-fill model. Therefore, upon receiving a first user request, the deployed first image editing model performs local editing on the user image to be processed based on the second prompt word in the first user request, obtaining the face image corresponding to the second prompt word, thereby effectively improving the replacement-style editing effect of the image.
[0108] Specifically, for scenarios requiring bangs generation or replacement, the bangs hairstyles in the first bangs images of multiple users are deleted to obtain bangs-free images. Based on the bangs-free images, second bangs images with various bangs hairstyles for each user are generated, resulting in a large number of bangs images paired with the bangs-free images. Furthermore, for each user, the user's bangs-free image and its mask are respectively stitched with the second bangs images and their masks for each bangs hairstyle, resulting in multiple first stitched images of the user with and without bangs. Based on each user's first stitched images and the first prompt word used to generate bangs hairstyles, the Lora model in the initial image editing model is fine-tuned, and the resulting first image editing model is deployed. The initial image editing model... The Lora model is embedded in a pre-trained Flux-fill model. Therefore, even with only a small number of images with and without bangs, the initial image editing model can be trained, enabling the model to accurately learn the generation of various bangs hairstyles from images without bangs. Furthermore, considering that the Lora model can extract bangs hairstyle features, after fine-tuning the Lora model, the image processing effect of the image editing model can be effectively improved compared to the conventional Flux-fill model. Therefore, upon receiving a first user request, the deployed first image editing model generates bangs hairstyles based on the second prompt word in the first user request and the user image to be processed, thereby obtaining the bangs image corresponding to the second prompt word, which effectively improves the generation effect of bangs hairstyles.
[0109] As an example, to better understand the solution in this application, we will illustrate it here with reference to the application scenario of notch generation, as follows:
[0110] The proposed solution mainly includes two stages: data preparation and preprocessing, and model training and inference, ultimately achieving automated image editing technology. The specific process is as follows:
[0111] (1) Data preparation and preprocessing
[0112] For an editing task that requires replacing a hairstyle without bangs with one with bangs, regardless of what kind of image the user uploads, the hairstyle without bangs in the user's image will be replaced with one with bangs, while other facial features, background content, etc., should not be changed.
[0113] After clarifying the specific task requirements, paired data (i.e., paired images) with and without bangs are collected. Considering the difficulty in obtaining paired data, this application collects some real images of thin, straight bangs, then uses a bangs removal model to remove the bangs hairstyle, and then uses the resulting images without bangs to generate m types of bangs images of arbitrary types. Both the bangs removal model and the bangs generation model are implemented using a self-developed model. Therefore, when n real images of thin, straight bangs are collected, n*(m+1) pairs of paired data can be obtained, where m represents the m types of bangs images of arbitrary types generated, and 1 represents an image without bangs.
[0114] (2) Model training and inference
[0115] like Figure 3 As shown in Figure (a), Figure 3 Figure (a) illustrates the training phase, where paired images are stitched together into a large image. The top left image is the original image (e.g., a generated image with any type of bangs or no bangs), and the top right image is the edited target image (e.g., a realistic image with thin bangs). The base model for the image editing model can use an open-source image generation model called Flux-fill, which is an image inpainting model.
[0116] The input to the image editing model includes not only the large image stitched together as described above, but also the corresponding image mask. Considering that while the base model of the image editing model has relatively good image editing capabilities, its performance is not particularly good in the vertical scenario of generating or changing bangs hairstyles, this application, based on an open-source image editing model, embeds a LoRa model for the bangs vertical category, thereby further training a LoRa model.
[0117] During training, corresponding prompts are also required, primarily to guide the model on specific image editing operations. After training is complete, inputting a notch-less image and the corresponding prompts allows the model to generate notches based on the input notch-less image and prompts during inference. Figure 3 As shown in Figure (b).
[0118] Using the embodiments of this application, even with a small number of samples for model training, the model can accurately achieve the purpose of generating bangs. Therefore, after deploying the image editing model, users only need to input an image without bangs and the corresponding prompt to quickly and accurately generate any image with bangs, reducing the burden of image editing for users.
[0119] Furthermore, the process described above in this application can be applied to other types of image local editing tasks to meet the diverse user needs mentioned above and provide users with more flexible and personalized image editing options.
[0120] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0121] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.
[0122] In one exemplary embodiment, such as Figure 4 As shown, an image processing apparatus is provided, including: a deletion module 402, a first generation module 404, a stitching module 406, a model adjustment module 408, and a second generation module 410, wherein:
[0123] The deletion module 402 is used to perform first feature removal processing on the first face images of multiple users to obtain the processed image;
[0124] The first generation module 404 is used to generate a second face image for each user, including multiple second features, based on the processed image; the first feature and the second feature are different morphological features under the same location or hair type.
[0125] The stitching module 406 is used to stitch together the user's processed image, the second face image and the corresponding image mask for each user to obtain multiple first stitched images for the user.
[0126] The model adjustment module 408 is used to fine-tune the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word for the generation of body parts or hair, and to deploy the resulting first image editing model; wherein, the initial image editing model is obtained by embedding the Lora model in the pre-trained Flux-fill model, and the Lora model is used to extract body part features or hair features.
[0127] The second generation module 410 is used to, upon receiving a first user request, perform local editing processing on the user image to be processed based on the second prompt word in the first user request using a deployed first image editing model, to obtain a face image corresponding to the second prompt word.
[0128] In one embodiment, the first feature and the second feature are different eyebrow shapes; or, the first feature and the second feature are different bangs hairstyles; or, the first feature and the second feature are different eye shapes; or, the first feature and the second feature are different mouth shapes.
[0129] In one embodiment, the first feature and the second feature are different bangs hairstyles. The first face image is the first bangs image, and the processed image is the image without bangs. The second face image is the second bangs image. The Lora model is used to extract bangs hairstyle features.
[0130] The deletion module 402 is also used to delete the bangs hairstyle in the first bangs image of multiple users to obtain a bangs-free image;
[0131] The first generation module 404 is also used to generate second bangs images of various bangs hairstyles for each user based on the bangs-free image;
[0132] The splicing module 406 is also used to splice the user's no-bangs image and no-bangs image mask with the second bangs image and second bangs image mask of each bangs hairstyle for each user, so as to obtain multiple first spliced images of the user with and without bangs.
[0133] The model adjustment module 408 is also used to fine-tune the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word used to generate the bangs hairstyle;
[0134] The second generation module 410 is further configured to generate a bangs hairstyle by using the deployed first image editing model to process the second prompt word in the first user request and the user image to be processed, thereby obtaining the bangs image corresponding to the second prompt word.
[0135] In one embodiment, the deletion module 402 is further configured to call the first model interface of the bangs removal model; and to input the first bangs images of each user in batches through the first model interface, so that the bangs removal model deletes the bangs hairstyle in the first bangs images of each user and outputs a bangs-free image.
[0136] In one embodiment, the first generation module 404 is further configured to obtain a third prompt word for generating various bangs hairstyles; call the second model interface of the bangs generation model; and input each user's bangs-free image and corresponding third prompt word in batches through the second model interface, so that the bangs generation model generates second bangs images of various bangs hairstyles for each user based on the bangs-free image and the third prompt word.
[0137] In one embodiment, the model adjustment module 408 is further configured to input the user's stitched image and a first prompt word for generating bangs hairstyles into the initial image editing model for each user, so that the initial image editing model performs bangs hairstyle generation processing based on the first prompt word and the stitched image to obtain predicted bangs images of various bangs hairstyles for the user; determine a loss value based on the predicted bangs image and a second bangs image in the stitched image; and fine-tune the Lora model in the initial image editing model based on the loss value so that the Lora model learns the bangs hairstyle features of each bangs hairstyle.
[0138] In one embodiment, the second generation module 410 is further configured to extract a second prompt word and a user image to be processed from the first user request; invoke the deployed first image editing model based on the first user request; input the second prompt word and the user image into the first image editing model, so that the first image editing model uses the fine-tuned parameters of the Lora model and the parameters of the Flux-fill model to perform bangs hairstyle generation processing on the user image to be processed by the second prompt word, thereby obtaining the bangs image corresponding to the second prompt word.
[0139] In the above embodiments, a first feature removal process is performed on the first face images of multiple users to obtain processed images; based on the processed images, a second face image of each user is generated, including multiple second features. The first and second features are different morphological features under the same location or hair type, thereby obtaining a large number of face images with and without the first and second features; furthermore, for each user, the processed image, the second face image, and the corresponding image mask are stitched together to obtain multiple first stitched images of the user; based on the first stitched images of each user and the first prompt words used for location or hair generation, the Lora model in the initial image editing model is fine-tuned, and the resulting first image editing model is deployed; wherein, the initial image editing model is a pre-trained Flux-fill model with an embedded Lora model. Therefore, the LoRa model is used to extract particulate or hair features. Thus, even with only a small number of face images containing only the first and second features, the initial image editing model can be trained, enabling the model to learn to generate images with either the first or second feature from images lacking them. Furthermore, considering that the LoRa model is used to extract particulate or hair features, fine-tuning the LoRa model can effectively improve the replacement-style editing effect compared to the conventional Flux-fill model. Therefore, upon receiving a first user request, the deployed first image editing model performs local editing on the user image to be processed based on the second prompt word in the first user request, obtaining the face image corresponding to the second prompt word, thereby effectively improving the replacement-style editing effect of the image.
[0140] Specifically, for scenarios requiring bangs generation or replacement, the bangs hairstyles in the first bangs images of multiple users are deleted to obtain bangs-free images. Based on the bangs-free images, second bangs images with various bangs hairstyles for each user are generated, resulting in a large number of bangs images paired with the bangs-free images. Furthermore, for each user, the user's bangs-free image and its mask are respectively stitched with the second bangs images and their masks for each bangs hairstyle, resulting in multiple first stitched images of the user with and without bangs. Based on each user's first stitched images and the first prompt word used to generate bangs hairstyles, the Lora model in the initial image editing model is fine-tuned, and the resulting first image editing model is deployed. The initial image editing model... The Lora model is embedded in a pre-trained Flux-fill model. Therefore, even with only a small number of images with and without bangs, the initial image editing model can be trained, enabling the model to accurately learn the generation of various bangs hairstyles from images without bangs. Furthermore, considering that the Lora model can extract bangs hairstyle features, after fine-tuning the Lora model, the image processing effect of the image editing model can be effectively improved compared to the conventional Flux-fill model. Therefore, upon receiving a first user request, the deployed first image editing model generates bangs hairstyles based on the second prompt word in the first user request and the user image to be processed, thereby obtaining the bangs image corresponding to the second prompt word, which effectively improves the generation effect of bangs hairstyles.
[0141] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0142] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements an image editing method.
[0143] In an exemplary embodiment, a computer device is provided, which may be a terminal, including a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0144] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0145] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image processing method described above.
[0146] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the image processing method described above.
[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the image processing method described above.
[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. An image processing method, characterized in that, The method includes: The first facial images of multiple users are processed by removing the first feature to obtain the processed image; Based on the processed image, a second face image of each user is generated, including multiple second features; the first feature and the second feature are different morphological features of the same location or hair type. For each user, the processed image of the user, the second face image, and the corresponding image mask are stitched together to obtain multiple first stitched images of the user; Based on the first stitched image of each user and the first prompt word for the generation of body parts or hair, the Lora model in the initial image editing model is fine-tuned, and the resulting first image editing model is deployed; wherein, the initial image editing model is obtained by embedding the Lora model in a pre-trained Flux-fill model, and the Lora model is used to extract body part features or hair features. Upon receiving a first user request, the first image editing model is deployed to perform local editing on the user image to be processed based on the second prompt word in the first user request, thereby obtaining the face image corresponding to the second prompt word.
2. The method according to claim 1, characterized in that, The first feature and the second feature are different eyebrow shapes; or, The first feature and the second feature are different bangs hairstyles; or, The first feature and the second feature are different eye shapes; or, The first feature and the second feature are different mouth shapes.
3. The method according to claim 1, characterized in that, The first feature and the second feature are different bangs hairstyles. The first face image is a first bangs image, the processed image is a no-bangs image, the second face image is a second bangs image, and the Lora model is used to extract bangs hairstyle features. The first feature removal process for the first face images of multiple users to obtain the processed image includes: deleting the bangs hairstyle in the first bangs images of multiple users to obtain an image without bangs; The process of generating second face images of each user, including multiple second features, based on the second face image includes: generating second bangs images of each user with multiple bangs hairstyles based on the image without bangs; The step of stitching together the processed image, the second face image, and the corresponding image mask of each user to obtain multiple first stitched images of the user includes: stitching together the user's image without bangs and the image mask without bangs with the second bangs image and the second bangs image mask of each bangs hairstyle to obtain multiple first stitched images of the user with and without bangs. The step of fine-tuning the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word for generating the bangs hairstyle includes: fine-tuning the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word for generating the bangs hairstyle; The step of performing local editing processing on the user image to be processed based on the second prompt word in the first user request to obtain the face image corresponding to the second prompt word includes: using the deployed first image editing model to perform bangs hairstyle generation processing on the second prompt word in the first user request and the user image to be processed to obtain the bangs image corresponding to the second prompt word.
4. The method according to claim 3, characterized in that, The step of deleting the bangs hairstyle from the first bangs images of multiple users to obtain bangs-free images includes: Call the first model interface of the notch removal model; The first model interface is used to input the first bangs images of each user in batches, so that the bangs removal model can delete the bangs hairstyle in the first bangs images of each user and output a bangs-free image.
5. The method according to claim 3, characterized in that, The second bangs image, generated based on the bangs-free image for each user with various bangs hairstyles, includes: Obtain third-party suggestions for generating various bangs hairstyles; Call the second model interface of the bangs generation model; By batch inputting the no-bangs images of each user and the corresponding third prompt words through the second model interface, the bangs generation model can generate second bangs images of various bangs hairstyles for each user based on the no-bangs images and the third prompt words.
6. The method according to claim 3, characterized in that, The step of fine-tuning the Lora model in the initial image editing model based on the stitched images of each user and the first prompt word used to generate the bangs hairstyle includes: For each user, the stitched image of the user and the first prompt word for generating bangs hairstyle are input into the initial image editing model, so that the initial image editing model performs bangs hairstyle generation processing based on the first prompt word and the stitched image to obtain predicted bangs images of various bangs hairstyles for the user; The loss value is determined based on the predicted bangs image and the second bangs image in the stitched image; The Lora model in the initial image editing model is fine-tuned based on the loss value so that the Lora model learns the bangs hairstyle features of each of the bangs hairstyles.
7. The method according to any one of claims 3 to 6, characterized in that, The step of generating a bangs hairstyle image by using the deployed first image editing model to process the second prompt word in the first user request and the user image to be processed, and obtaining the bangs image corresponding to the second prompt word, includes: Extract the second prompt word and the user image to be processed from the first user request; The first image editing model is invoked based on the first user request; The second prompt word and the user image are input into the first image editing model, so that the first image editing model uses the fine-tuned parameters of the Lora model and the parameters of the Flux-fill model to generate a bangs hairstyle for the user image to be processed based on the second prompt word, thereby obtaining the bangs image corresponding to the second prompt word.
8. An image processing apparatus, characterized in that, The device includes: The deletion module is used to remove the first feature from the first face images of multiple users to obtain the processed image; The first generation module is used to generate a second face image for each user, including multiple second features, based on the processed image; the first feature and the second feature are different morphological features of the same location or hair type. The stitching module is used to stitch together the processed image, the second face image, and the corresponding image mask of each user to obtain multiple first stitched images of the user. The model adjustment module is used to fine-tune the Lora model in the initial image editing model based on the first stitched image of each user and the first prompt word for the generation of body parts or hair, and to deploy the resulting first image editing model; wherein, the initial image editing model is obtained by embedding the Lora model in a pre-trained Flux-fill model, and the Lora model is used to extract body part features or hair features. The second generation module is used to, upon receiving a first user request, perform local editing processing on the user image to be processed based on the second prompt word in the first user request using the deployed first image editing model, to obtain the face image corresponding to the second prompt word.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.