Image processing method, image processing model training method, and related device

Abstract

The application discloses an image processing method, an image processing model training method and related equipment; the original image to be processed and style description information of a target image style can be acquired; scene description information of the original image is generated; face recognition processing is performed on an object contained in the original image to obtain face attribute information of the object in the original image; the original image is subjected to diffusion processing according to the style description information, the scene description information and the face attribute information, so as to generate a target image corresponding to the original image under the target image style. The application can make the generated stylized target image better integrate original image features, be closer to the original image, retain some important information of the original image, and greatly improve the quality of the generated image.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an image processing method, an image processing model training method, and related equipment. Background Technology

[0002] With the development of deep learning, image processing technology is being applied to an increasing number of fields and products. For example, image processing technology can be used for image stylization. Image stylization specifically refers to transforming an image into a new image with a unique artistic style through specific algorithms and techniques. For instance, image stylization can convert a real photograph into an artistically styled image; or it can transform an ID photo into a cartoon-style image. Specifically, image stylization can be applied to visual enhancement, brand advertising design, and personalized user content generation. This diverse and engaging approach can inject more vitality into multimedia communities, increase creativity and fun, improve content quality, and promote dissemination and interaction.

[0003] Currently, the techniques for image stylization have shortcomings in image detail processing. This often leads to significant differences in features between the generated stylized image and the original image, and the original style cannot be well integrated with the target style, resulting in poor quality of the generated stylized image. Summary of the Invention

[0004] This application provides an image processing method, an image processing model training method, and related equipment. The related equipment may include an image processing device, an image processing model training device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the quality of the generated image.

[0005] This application provides an image processing method, including:

[0006] Obtain style description information for the original and target images to be processed;

[0007] The scene description information for generating the original image;

[0008] Facial recognition processing is performed on the objects contained in the original image to obtain the facial attribute information of the objects in the original image;

[0009] Based on the style description information, the scene description information, and the facial attribute information, the original image is subjected to diffusion processing to generate a target image corresponding to the original image under the target image style.

[0010] This application provides an image processing model training method, including:

[0011] Acquire training data and an image processing model, wherein the training data includes sample images under the sample image style and style description information of the sample image style;

[0012] The scene description information of the sample image is generated;

[0013] Facial recognition processing is performed on the objects contained in the sample image to obtain the facial attribute information of the objects in the sample image;

[0014] The sample image is subjected to forward diffusion processing to obtain the processed sample image;

[0015] Using the image processing model, the processed sample image is subjected to reverse diffusion processing based on the style description information, the scene description information, and the facial attribute information to obtain a stylized sample image.

[0016] Based on the sample image and the stylized sample image, the parameters of the image processing model are adjusted to obtain the trained image processing model.

[0017] Accordingly, embodiments of this application provide an image processing apparatus, including:

[0018] The first acquisition unit is used to acquire style description information of the original image to be processed and the target image style;

[0019] The first generation unit is used to generate scene description information of the original image;

[0020] The first recognition unit is used to perform facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image.

[0021] The first diffusion unit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate a target image corresponding to the original image under the target image style.

[0022] Accordingly, embodiments of this application provide an image processing model training apparatus, comprising:

[0023] The second acquisition unit is used to acquire training data and an image processing model. The training data includes sample images under the sample image style and style description information of the sample image style.

[0024] The second generation unit is used to generate scene description information for the sample image;

[0025] The second recognition unit is used to perform facial recognition processing on the objects contained in the sample image to obtain facial attribute information of the objects in the sample image.

[0026] The second diffusion unit is used to perform forward diffusion processing on the sample image to obtain a processed sample image.

[0027] The third diffusion unit is used to perform reverse diffusion processing on the processed sample image based on the style description information, the scene description information and the facial attribute information through the image processing model to obtain the stylized sample image;

[0028] The adjustment unit is used to adjust the parameters of the image processing model based on the sample image and the stylized sample image to obtain the trained image processing model.

[0029] Optionally, in some embodiments of this application, the first diffusion unit may include a diffusion subunit and a face adjustment subunit, as follows:

[0030] The diffusion subunit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate an initial stylized image corresponding to the original image under the target image style;

[0031] The face adjustment subunit is used to perform object face adjustment processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0032] Optionally, in some embodiments of this application, the face adjustment subunit may be specifically used to crop the object's face region of the initial stylized image to obtain an object's face image; to redraw the object's face image to obtain a redrawn object's face image; and to fuse the redrawn object's face image with the initial stylized image based on the position information of the object's face region in the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0033] Optionally, in some embodiments of this application, the step "redrawing the object's facial image to obtain a redrawn object's facial image" may include:

[0034] Based on the initial stylized image and the positional information of the object's facial region in the initial stylized image, an initial mask image is generated, the initial mask image including masked regions and non-masked regions;

[0035] The edges between the masked and non-masked regions in the initial mask image are blurred to obtain the target mask image;

[0036] Based on the target mask image, the object's facial image is subjected to diffusion processing to obtain a redrawn object's facial image.

[0037] Optionally, in some embodiments of this application, the first diffusion unit may include a diffusion subunit and a limb adjustment subunit, as follows:

[0038] The diffusion subunit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate an initial stylized image corresponding to the original image under the target image style;

[0039] The limb adjustment subunit is used to perform object limb adjustment processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0040] Optionally, in some embodiments of this application, the limb adjustment subunit may be specifically used to perform cropping processing on the object limb region of the initial stylized image to obtain a limb image; to perform redrawing processing on the limb image to obtain a redrawn limb image; and to perform fusion processing on the redrawn limb image and the initial stylized image based on the position information of the object limb region in the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0041] Optionally, in some embodiments of this application, the step "redrawing the limb image to obtain a redrawn limb image" may include:

[0042] Based on the initial stylized image and the positional information of the object's limb region in the initial stylized image, an initial mask image is generated, the initial mask image including masked regions and non-masked regions;

[0043] The edges between the masked and non-masked regions in the initial mask image are blurred to obtain the target mask image;

[0044] Based on the target mask image, the limb image is subjected to diffusion processing to obtain the redrawn limb image.

[0045] Optionally, in some embodiments of this application, the first diffusion unit may include a prompting determination subunit, a noise prediction subunit, and a denoising subunit, as follows:

[0046] The prompt determination subunit is used to determine prompt information based on the style description information, the scene description information, and the facial attribute information;

[0047] A noise prediction subunit is used to predict image noise information of the original image based on the prompt information;

[0048] The denoising subunit is used to perform denoising processing on the original image based on the image noise information to obtain the target image corresponding to the original image under the target image style.

[0049] Optionally, in some embodiments of this application, the noise prediction subunit may be specifically used to: when a preset text unit is identified from the prompt information, search for the target embedding feature corresponding to the preset text unit from a feature mapping set, the feature mapping set including mapping relationship information between text units and embedding features; fuse the target embedding feature with the prompt information to obtain processed prompt information; perform semantic feature extraction processing on the processed prompt information to obtain prompt semantic features; and predict the image noise information of the original image based on the prompt semantic features.

[0050] Optionally, in some embodiments of this application, the noise prediction subunit may be used to obtain the weight adjustment parameters corresponding to the style description information; fuse the weight adjustment parameters with the weight parameters of the image processing model to obtain the fused weight parameters; and, based on the fused weight parameters, perform attention feature interaction processing on the prompt information and the original image through the image processing model to predict the image noise information of the original image.

[0051] Optionally, in some embodiments of this application, the step "fusing the weight adjustment parameters with the weight parameters of the image processing model to obtain fused weight parameters" may include:

[0052] Based on the feature retention intensity information of the original image, the fusion intensity information of the weight adjustment parameters is determined;

[0053] Based on the fusion intensity information, the weight adjustment parameters are fused with the weight parameters of the image processing model to obtain the fused weight parameters.

[0054] Optionally, in some embodiments of this application, the first diffusion unit may include a diffusion subunit and an image quality restoration subunit, as follows:

[0055] The diffusion subunit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate an initial stylized image corresponding to the original image under the target image style;

[0056] The image quality restoration subunit is used to perform image quality restoration processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0057] Optionally, in some embodiments of this application, the first diffusion unit may include a determination subunit, a feature extraction subunit, and a first diffusion processing subunit, as follows:

[0058] The determining subunit is used to determine a reference image containing the target pose;

[0059] A feature extraction subunit is used to extract pose feature information of the reference image;

[0060] The first diffusion processing subunit is used to perform diffusion processing on the original image based on the style description information, the scene description information, the facial attribute information and the pose feature information, so as to generate a target image corresponding to the original image under the target image style and the target pose.

[0061] Optionally, in some embodiments of this application, the first diffusion unit may include an image acquisition subunit, a prompt information determination subunit, and a second diffusion processing subunit, as follows:

[0062] The image acquisition subunit is used to acquire a reference image under the target image style and extract feature information from the reference image;

[0063] The prompt information determination subunit is used to determine the prompt information based on the style description information, the scene description information, and the facial attribute information;

[0064] The second diffusion processing subunit is used to perform diffusion processing on the original image based on the feature information of the reference image and the prompt information, so as to generate a target image corresponding to the original image under the target image style.

[0065] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0066] An electronic device provided in this application includes a processor and a memory. The memory stores multiple instructions, and the processor loads the instructions to execute the steps in the image processing method and image processing model training method provided in this application.

[0067] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps in the image processing method and image processing model training method provided in this application.

[0068] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the image processing method and image processing model training method provided in embodiments of this application.

[0069] This application provides an image processing method, an image processing model training method, and related equipment, which can acquire style description information of the original image and the target image style; generate scene description information of the original image; perform facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image; and perform diffusion processing on the original image according to the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style.

[0070] This application can not only utilize style description information, but also combine scene description information and facial attribute information of objects in the original image to perform image stylization processing. This allows the generated stylized target image to better integrate the features of the original image, be closer to the original image, retain some important information of the original image, and greatly improve the quality of the generated image. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1a This is a schematic diagram of a scene illustrating the image processing method provided in an embodiment of this application;

[0073] Figure 1b This is a flowchart of the image processing method provided in the embodiments of this application;

[0074] Figure 1cThese are illustrative diagrams illustrating the image processing method and image processing model training method provided in the embodiments of this application;

[0075] Figure 1d This is a model architecture diagram of the image processing method and image processing model training method provided in the embodiments of this application;

[0076] Figure 1e This is another illustrative diagram of the image processing method and image processing model training method provided in the embodiments of this application;

[0077] Figure 1f This is another illustrative diagram of the image processing method and image processing model training method provided in the embodiments of this application;

[0078] Figure 1g This is another flowchart of the image processing method and image processing model training method provided in the embodiments of this application;

[0079] Figure 1h This is another model architecture diagram of the image processing method and image processing model training method provided in the embodiments of this application;

[0080] Figure 1i This is another flowchart of the image processing method and image processing model training method provided in the embodiments of this application;

[0081] Figure 2 This is another flowchart of the image processing model training method provided in the embodiments of this application;

[0082] Figure 3a This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0083] Figure 3b This is a schematic diagram of the structure of the image processing model training device provided in the embodiments of this application;

[0084] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0085] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0086] This application provides an image processing method, an image processing model training method, and related equipment. The related equipment may include an image processing device, an image processing model training device, an electronic device, a computer-readable storage medium, and a computer program product.

[0087] The image processing device can be integrated into an electronic device, such as a terminal or server.

[0088] It is understood that the image processing method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0089] In one embodiment, such as Figure 1a As shown, an image processing method is executed jointly by a terminal and a server, as an example. The image processing system provided in this application includes a terminal 10 and a server 11, etc.; the terminal 10 and the server 11 are connected via a network, such as a wired or wireless network, etc., wherein the image processing device can be integrated into the server.

[0090] Server 11 can be used to: acquire style description information of the original image and the target image style; generate scene description information of the original image; perform facial recognition processing on objects contained in the original image to obtain facial attribute information of the objects in the original image; and perform diffusion processing on the original image according to the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style. Server 11 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0091] Terminal 10 can be used to: send style description information of the original image to be processed and the target image style to server 11, so that server 11 can perform image stylization processing on the original image based on the style description information to obtain the target image; terminal 10 can also receive the target image sent by server 11. Terminal 10 may include a mobile phone, vehicle terminal, aircraft, tablet computer, laptop computer, or personal computer (PC), etc. A client can also be set on terminal 10, which can be an application client or a browser client, etc.

[0092] The image processing and other steps in the aforementioned server 11 can also be performed by the terminal 10.

[0093] The image processing model training device can be integrated into an electronic device, such as a terminal or a server. The image processing model training method in this embodiment can be executed on a server, or it can be executed jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0094] In one embodiment, an image processing model training method is implemented jointly by a terminal and a server. The image processing model training system provided in this application includes a terminal and a server, which are connected via a network, such as a wired or wireless network. The image processing model training device can be integrated into the server.

[0095] The server can be used to: acquire training data and an image processing model, wherein the training data includes sample images under a sample image style and style description information of the sample image style; generate scene description information of the sample images; perform facial recognition processing on objects contained in the sample images to obtain facial attribute information of the objects in the sample images; perform forward diffusion processing on the sample images to obtain processed sample images; perform reverse diffusion processing on the processed sample images using the image processing model, based on the style description information, the scene description information, and the facial attribute information, to obtain stylized sample images; and adjust the parameters of the image processing model based on the sample images and the stylized sample images to obtain a trained image processing model. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0096] The terminal can be used to receive a trained image processing model sent by the server, the trained image processing model being used for image stylization processing. The terminal can include a mobile phone, vehicle-mounted terminal, aircraft, tablet computer, laptop computer, or personal computer (PC), etc. A client can also be set on the terminal, which can be an application client or a browser client, etc.

[0097] The image processing methods and image processing model training methods provided in the embodiments of this application relate to machine learning and computer vision technologies in the field of artificial intelligence.

[0098] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0099] This embodiment will be described from the perspective of an image processing device, which can be integrated into an electronic device, such as a server or a terminal.

[0100] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0101] like Figure 1b As shown, the specific process of this image processing method can be as follows:

[0102] 101. Obtain style description information for the original image and the target image to be processed.

[0103] The original image is the image that needs to undergo image stylization processing. The target image style can be artistic, ACG, oil painting, abstract, or ink painting, etc., and this embodiment does not limit this. The image style in this embodiment can refer to the overall feeling that an image gives in terms of sharpness, contrast, content, hue, texture, and shape. Here, ACG (Animation, Comics, Games) refers to the styles of some anime, games, and two-dimensional culture.

[0104] Specifically, this style description information can be used to guide the model on how to transform the original image into an image in the style of the target image. The style description information provides the model with explicit semantic guidance, helping it understand the user's desired stylization effect. Specifically, the style description information can be textual descriptions of the target image's style.

[0105] 102. Generate scene description information for the original image.

[0106] The scene description information may include the number of objects in the image, object information, environmental features in the image, scene atmosphere, main colors of the scene, object behavior in the image, and may even include clothing features and physical characteristics of objects in the image. This embodiment does not impose any restrictions on this.

[0107] In a specific embodiment, image-to-text (IPM) models can be used to perform scene recognition on the original image and generate scene description information. Image-to-text (IPM), also known as image tagging, refers to the use of computer vision and natural language processing technologies to enable computers to understand image content and generate descriptive text using natural language.

[0108] Specifically, the graph-to-text model is an image-to-text model that can generate corresponding text descriptions based on the input image. Graph-to-text models can be generative adversarial networks, variational autoencoders, diffusion models, Transformer architecture models, etc., and this embodiment does not impose any restrictions on them.

[0109] Specifically, the image-to-text model can include an image encoder and a text generator. The image encoder can extract image feature information from the original image, and the extracted image feature information can be used as input to the text generator to guide the text generation process. The text generator can then generate text step by step based on the image feature information and preset text prompts.

[0110] 103. Perform facial recognition processing on the objects contained in the original image to obtain the facial attribute information of the objects in the original image.

[0111] The objects contained in the original image can be people, animals, etc.

[0112] Facial attribute information can include age, gender, expression, hairstyle, and facial features. Facial features can include the position and shape of the eyes, nose, and mouth.

[0113] Specifically, facial recognition models can be used to perform facial recognition processing on objects contained in the original image. These facial recognition models can employ various types of neural networks, such as Visual Geometry Group Network (VGGNet), Residual Network (ResNet), and Dense Convolutional Network (DenseNet), among others.

[0114] 104. Based on the style description information, the scene description information, and the facial attribute information, perform diffusion processing on the original image to generate a target image corresponding to the original image under the target image style.

[0115] Specifically, the diffusion processing here is reverse diffusion processing, which is a process of gradually removing noise from the image to generate a clear image that meets the requirements.

[0116] Specifically, an image processing model can be used to perform diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style. This image processing model can specifically be a diffusion model.

[0117] In machine learning, diffusion models, or diffusion probability models, are a type of latent variable model. They are Markov chains trained using variational estimation. The goal of diffusion models is to learn the latent structure of a dataset by modeling how data points diffuse in the latent space. In computer vision, this means training a neural network to learn the inverse diffusion process, enabling it to denoise images with superimposed Gaussian noise.

[0118] Optionally, in this embodiment, the step of "performing diffusion processing on the original image according to the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style" may include:

[0119] The prompt information is determined based on the style description information, the scene description information, and the facial attribute information;

[0120] Based on the prompt information, predict the image noise information of the original image;

[0121] Based on the image noise information, the original image is denoised to obtain the target image corresponding to the original image under the target image style.

[0122] In some embodiments, style description information, scene description information, and facial attribute information can be directly concatenated to obtain the prompt information. In other embodiments, style description information, scene description information, and facial attribute information can be filled into the corresponding positions in a preset prompt template to obtain the filled preset prompt information. The filled preset prompt information can then serve as the overall prompt information, thereby guiding the noise reduction process. The preset prompt template can be: Style: [], Scene: [], Face: [], etc.

[0123] Specifically, the denoising process can include multiple steps, such as a T-step denoising process. In each denoising step, the noise contained in the image is predicted by the image processing model. The predicted noise information of the current image is used to gradually remove noise from the features, thereby obtaining a stylized image after T-step denoising.

[0124] Specifically, in this embodiment, the step "predicting image noise information of the original image based on the prompt information" may include:

[0125] Based on the aforementioned prompt information, the original image is downsampled at multiple scales to obtain target features;

[0126] The target features are upsampled at multiple scales to obtain the image noise information of the original image.

[0127] In some embodiments, the step "based on the prompt information, performing downsampling processing on the original image at multiple scales to obtain target features" may include:

[0128] Extract the semantic features of the prompt information and the image features of the original image respectively;

[0129] Attention feature interaction processing is performed on the semantic feature information and the image feature information to obtain attention features;

[0130] The attention features are downsampled to obtain downsampled features;

[0131] The downsampled features are then identified as new image feature information.

[0132] Return to the step of performing attention feature interaction processing on the semantic feature information and the image feature information to obtain attention features, until the number of iterations meets the preset condition and the target feature is obtained.

[0133] In this embodiment, an attention mechanism can be used to guide the model to focus on important parts of the input data, helping the model understand which features are related to the description of the prompt information, thereby improving the generation quality.

[0134] The preset conditions can be set according to the actual situation. The total number of iterations can be determined based on the number of downsampling modules in the encoding part of the image processing model.

[0135] In some embodiments, the step "upsampling the target features at multiple scales to obtain image noise information of the original image" may include:

[0136] The target features are upsampled to obtain the upsampled features;

[0137] Based on the prompt information, attention processing is performed on the upsampled features to obtain attention-processed features;

[0138] The attention processing features are used as new target features;

[0139] Return to the step of upsampling the target features to obtain the upsampled features, until the number of iterations meets the preset condition, and obtain the image noise information of the original image.

[0140] The preset conditions can be set according to the actual situation. The total number of iterations can be determined based on the number of upsampling modules in the decoding part of the image processing model.

[0141] The upsampling process progressively generates higher-resolution image features. An attention mechanism is integrated throughout both the downsampling and upsampling processes, allowing cue information to be incorporated into the image features. Simultaneously, through the attention mechanism, the model can refer to the cue information at each denoising step, ensuring that the generated image matches the cue information.

[0142] Specifically, the step "denoising the original image based on the image noise information to obtain the target image corresponding to the original image under the target image style" may include:

[0143] Based on the image noise information, the original image is denoised to obtain a denoised feature map;

[0144] The denoised feature map is used as the new original image;

[0145] Return to the step of predicting the image noise information of the original image based on the prompt information, until the number of iterations meets the preset condition, and obtain the target image corresponding to the original image under the target image style.

[0146] In each round of denoising, the current image noise information can be predicted by the image processing model to gradually remove noise from the image and generate a stylized image.

[0147] Optionally, in this embodiment, the step of "performing diffusion processing on the original image according to the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style" may include:

[0148] Based on the style description information, the scene description information, and the facial attribute information, the original image is subjected to diffusion processing to generate an initial stylized image corresponding to the original image under the target image style;

[0149] The initial stylized image is subjected to object facial adjustment processing to obtain the target image corresponding to the original image under the target image style.

[0150] In this embodiment, a facial effect optimization strategy is introduced, which can adjust the face of the object in the initial stylized image generated by the image processing model.

[0151] The step "performing facial adjustment processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style" may include:

[0152] Detect the proportion of the object's facial region in the initial stylized image;

[0153] When the proportion information is less than a preset value, facial adjustment processing is performed on the facial region of the object in the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0154] The preset value can be set according to the actual situation, such as 20%. If the proportion of the object's facial region is not less than the preset value, there is no need to adjust the object's face in the initial stylized image, and the initial stylized image can be directly determined as the target image. Because a small face results in a low proportion of the object's facial region in the image, the network's attention to the face is weaker during inference, which may lead to the generated result having an illogical distribution of facial features and poor quality. This embodiment addresses the problem of small faces by adjusting the facial region of the object whose proportion is less than the preset value, making the irrelevant distribution of the face in the target image more consistent with common sense and improving the quality of the generated image.

[0155] Optionally, in this embodiment, the step "performing object facial adjustment processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style" may include:

[0156] The initial stylized image is cropped to obtain the object's facial image;

[0157] The object's facial image is redrawn to obtain a redrawn object's facial image.

[0158] Based on the positional information of the object's facial region in the initial stylized image, the redrawn object's facial image is fused with the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0159] Specifically, the redrawn object's facial image is fused with the initial stylized image, which can be done by overlaying the redrawn object's facial image onto the initial stylized image.

[0160] In this embodiment, the facial region of the object in the initial stylized image can be cropped out and redrawn. This ensures that the network's attention is entirely focused on the facial region during the redrawing process. The redrawn result is then warped back to the original image position. Specifically, the redrawn facial image can be overlaid and pasted onto the original position of the facial region in the initial stylized image.

[0161] In some embodiments, the above-described image processing model can be used to redraw the object's facial image; in other embodiments, other models can also be used to redraw the object's facial image, and this embodiment does not limit this.

[0162] Optionally, in this embodiment, the step "redrawing the object's facial image to obtain a redrawn object's facial image" may include:

[0163] Based on the initial stylized image and the positional information of the object's facial region in the initial stylized image, an initial mask image is generated, the initial mask image including masked regions and non-masked regions;

[0164] The edges between the masked and non-masked regions in the initial mask image are blurred to obtain the target mask image;

[0165] Based on the target mask image, the object's facial image is subjected to diffusion processing to obtain a redrawn object's facial image.

[0166] In this embodiment, in order to ensure that the redrawn object's facial image blends with the original image (i.e., the initial stylized image), this embodiment can combine a mask with edge transitions to redraw the object's facial image, thus obtaining a harmonious result.

[0167] Specifically, the initial mask image is a binary mask, where pixels with a value of 1 represent masked areas, indicating regions that need to be redrawn; pixels with a value of 0 represent non-masked areas, indicating regions that retain the original image. Specifically, the initial mask image has the same size as the initial stylized image, and the position of the masked areas in the initial mask image is determined based on the position of the object's facial region in the initial stylized image; the two positions correspond, and the size of the masked areas is the same as the size of the object's facial region.

[0168] The blurring of region edges can be achieved by transitioning the edges from 0 to 1, or from 1 to 0. Specifically, this blurring can involve smoothing the pixel values ​​of some pixels in the mask region of the initial mask image that are close to the non-masked region, so that the pixel values ​​of these pixels are between 0 and 1. This results in a mask with edge transitions—the target mask image. Redrawing the region based on the target mask image can achieve a natural blending of the redrawn region with the original image boundary, avoiding obvious boundaries between the redrawn region and the preserved region.

[0169] Specifically, the diffusion processing of the object's facial image based on the target mask image can utilize the image processing model used to generate the initial stylized image in the previous embodiments. The target mask image can serve as a cue for the diffusion processing to guide the redrawing process.

[0170] In specific scenarios, during image stylization, the generated stylized images may contain various flaws, such as limb distortion. This embodiment can use the text inversion method to mitigate these flaws to some extent. Text inversion learns pseudo-words in the embedding space of a text-to-image model to represent new concepts, and then uses natural language sentences composed of these pseudo-words to guide personalized generation. Text inversion is a lightweight fine-tuning method suitable for quickly customizing specific needs in generation tasks.

[0171] Optionally, in this embodiment, the step of "predicting image noise information of the original image based on the prompt information" may include:

[0172] When a preset text unit is identified from the prompt information, the target embedding feature corresponding to the preset text unit is searched from the feature mapping set, which includes the mapping relationship information between the text unit and the embedding feature;

[0173] The target embedded features are fused with the prompt information to obtain the processed prompt information;

[0174] The processed prompt information is subjected to semantic feature extraction to obtain prompt semantic features;

[0175] Based on the semantic features of the prompts, predict the image noise information of the original image.

[0176] Specifically, the preset text units in the prompt information can be presented as placeholder identifiers, which can be used to represent new concepts. When the system processes the input prompt information, it scans the special symbols in the text corresponding to the prompt information, such as < and >, and identifies the content between these symbols as placeholders. In this embodiment, the learned embedded features corresponding to the placeholder identifiers can be obtained through a feature mapping set.

[0177] Specifically, the target embedding features are fused with the prompt information. This can be done by replacing the corresponding preset text units in the prompt information with the target embedding features, that is, the target embedding features will replace the original words in the prompt information.

[0178] Specifically, the feature mapping set can include the mapping relationship between multiple placeholder identifiers and their embedded features. The embedded feature corresponding to each placeholder identifier is obtained through training. Specifically, a set of images related to the placeholder identifiers can be acquired first, and a model can be trained based on these images to learn an embedded feature for the placeholder identifier. Through the optimization process, the embedded feature of the placeholder identifier will be continuously adjusted so that it can accurately represent the new concept in the latent space.

[0179] In image stylization, limb distortion is a common problem, especially when generating images with complex poses or specific styles. In this embodiment, text inversion can be used to avoid hand and limb distortion. For example, preset text units such as "Avoid limb distortion" can be set in the prompt message. When the system recognizes this preset text unit in the prompt message, it can obtain its corresponding target embedding features. These target embedding features are then used to replace the preset text unit in the prompt message, resulting in an updated prompt message. This updated prompt message then guides the generation of the stylized image. However, the effectiveness of this text inversion method is limited. This application can also introduce a limb adjustment process to further avoid the problem of limb distortion.

[0180] Optionally, in this embodiment, the step of "performing diffusion processing on the original image according to the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style" may include:

[0181] Based on the style description information, the scene description information, and the facial attribute information, the original image is subjected to diffusion processing to generate an initial stylized image corresponding to the original image under the target image style;

[0182] The initial stylized image is subjected to object limb adjustment processing to obtain the target image corresponding to the original image under the target image style.

[0183] In this embodiment, a limb effect optimization strategy is introduced, which can adjust the limbs of objects in the initial stylized image generated by the image processing model.

[0184] Optionally, in this embodiment, the step "adjusting the object's limbs in the initial stylized image to obtain the target image corresponding to the original image under the target image style" may include:

[0185] The initial stylized image is cropped to obtain a limb image;

[0186] The limb image is redrawn to obtain a redrawn limb image;

[0187] Based on the positional information of the object's limb region in the initial stylized image, the redrawn limb image is fused with the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0188] Specifically, the redrawn limb image is fused with the initial stylized image, which can be done by overlaying the redrawn limb image onto the initial stylized image.

[0189] In this embodiment, the object's limb region can be cropped from the initial stylized image and redrawn. This ensures that the network's attention is entirely focused on the object's limb region during the redrawing process. The redrawn result is then warped back to the original image position. Specifically, the redrawn limb image can be overlaid and pasted onto the original position of the object's limb region in the initial stylized image.

[0190] In some embodiments, the above-described image processing model can be used to redraw the limb image; in other embodiments, other models can also be used to redraw the limb image, and this embodiment does not limit this.

[0191] Optionally, in this embodiment, the step "redrawing the limb image to obtain a redrawn limb image" may include:

[0192] Based on the initial stylized image and the positional information of the object's limb region in the initial stylized image, an initial mask image is generated, the initial mask image including masked regions and non-masked regions;

[0193] The edges between the masked and non-masked regions in the initial mask image are blurred to obtain the target mask image;

[0194] Based on the target mask image, the limb image is subjected to diffusion processing to obtain the redrawn limb image.

[0195] In this embodiment, in order to ensure that the boundary between the redrawn limb image and the original image (i.e., the initial stylized image) is blended, this embodiment can combine a mask with edge transitions to redraw the limb image, thus obtaining a harmonious result.

[0196] Specifically, the initial mask image is a binary mask, where pixels with a value of 1 represent masked areas, indicating regions that need to be redrawn; pixels with a value of 0 represent non-masked areas, indicating regions that retain the original image. Specifically, the initial mask image has the same size as the initial stylized image, and the position of the masked areas in the initial mask image is determined based on the position of the object's limb region in the initial stylized image; the two positions correspond, and the size of the masked areas is the same as the size of the object's limb region.

[0197] The blurring of region edges can be achieved by transitioning the edges from 0 to 1, or from 1 to 0. Specifically, this blurring can involve smoothing the pixel values ​​of some pixels in the mask region of the initial mask image that are close to the non-masked region, so that the pixel values ​​of these pixels are between 0 and 1. This results in a mask with edge transitions—the target mask image. Redrawing the region based on the target mask image can achieve a natural blending of the redrawn region with the original image boundary, avoiding obvious boundaries between the redrawn region and the preserved region.

[0198] Specifically, the diffusion processing of the limb image based on the target mask image can utilize the image processing model used to generate the initial stylized image in the previous embodiments. The target mask image can serve as a cue for the diffusion processing to guide the redrawing process.

[0199] In specific scenarios, when the target image style is an image style that is not supported by the original base model of the image processing model, style customization is required. Specifically, weight adjustment parameters corresponding to the target image style can be introduced. These weight adjustment parameters are low-rank matrices, i.e., LoRa small models. Through these low-rank matrices, the weight parameters of the image processing model can be adjusted, thereby affecting the generation of the image.

[0200] Lora (Low-Rank Adaptation) is a parameter tuning method for deep learning models. It adapts to new tasks or data by adding a low-rank matrix to a pre-trained model, thereby achieving rapid model adaptation and efficient training.

[0201] After pre-training the image processing model, a small amount of data can be collected, weight adjustment parameters can be introduced, and customized style effects can be achieved through model fine-tuning. Specifically, this image processing method also includes:

[0202] Acquire training sample data and preset weight adjustment parameters for the target style task. The training sample data includes sample reference images for the target style and style description information for the target style.

[0203] The weight adjustment parameters are fused with the weight parameters of the image processing model to obtain the fused weight parameters;

[0204] The sample reference image is subjected to noise addition processing to obtain a noisy sample reference image;

[0205] Generate reference scene description information for the sample reference image; and perform facial recognition processing on the objects in the sample reference image to obtain reference facial attribute information of the objects in the sample reference image;

[0206] Using the image processing model, based on the fused weight parameters, the noisy sample reference image is denoised using the style description information of the target style, the reference scene description information, and the reference facial attribute information to obtain the denoised sample reference image.

[0207] The weight adjustment parameters are updated based on the sample reference image and the denoised sample reference image to obtain the updated weight adjustment parameters.

[0208] This weight adjustment parameter is also known as the style adjustment adaptive parameter.

[0209] Specifically, the target style task can be a task that is not supported by the original primitive model of the image processing model, that is, the original primitive model of the image processing model cannot convert the image into an image under the target style.

[0210] The fusion of the weight adjustment parameters with the weight parameters of the image processing model can be achieved by adding the weight adjustment parameters to the weight parameters of the image processing model. Specifically, the weight adjustment parameters can be fused with the weight parameters of a specific layer (such as the attention layer) in the image processing model.

[0211] Specifically, the weight adjustment parameters can be updated based on the similarity between the sample reference image and the denoised sample reference image. This optimizes the weight adjustment parameters so that the denoised sample reference image closely approximates the original sample reference image, resulting in the updated weight adjustment parameters. In particular, the similarity between the denoised sample reference image and the original sample reference image can be greater than a preset value, which can be set according to actual conditions.

[0212] Optionally, in this embodiment, the step of "predicting image noise information of the original image based on the prompt information" may include:

[0213] Obtain the weight adjustment parameters corresponding to the style description information;

[0214] The weight adjustment parameters are fused with the weight parameters of the image processing model to obtain the fused weight parameters.

[0215] Using the image processing model, based on the fused weight parameters, attention feature interaction processing is performed on the prompt information and the original image to predict the image noise information of the original image.

[0216] Different weight adjustment parameters can correspond to different image styles. In this embodiment, the corresponding weight adjustment parameters can be obtained according to the target image style indicated in the style description information.

[0217] The fusion of the weight adjustment parameters with the weight parameters of the image processing model can be achieved by adding the weight adjustment parameters to the weight parameters of the image processing model. Specifically, the weight adjustment parameters can be fused with the weight parameters of a specific layer (such as the attention layer) in the image processing model.

[0218] Specifically, during online inference, if a user needs to retain more features of the original image, they can adjust the fusion strength information of the weight adjustment parameters, that is, adjust the fusion strength of LoRa to control the retention strength of the original image features.

[0219] Optionally, in this embodiment, the step "fusing the weight adjustment parameters with the weight parameters of the image processing model to obtain fused weight parameters" may include:

[0220] Based on the feature retention intensity information of the original image, the fusion intensity information of the weight adjustment parameters is determined;

[0221] Based on the fusion intensity information, the weight adjustment parameters are fused with the weight parameters of the image processing model to obtain the fused weight parameters.

[0222] Specifically, the fusion strength information is the weight information corresponding to the weight adjustment parameters. Based on this weight information, the weight adjustment parameters and the weight parameters of the image processing model can be weighted and fused to obtain the fused weight parameters.

[0223] The greater the feature retention strength, the smaller the fusion strength of the weight adjustment parameters; conversely, the smaller the feature retention strength, the greater the fusion strength of the weight adjustment parameters.

[0224] Specifically, this application can also set up an image quality restoration process to address the current characteristics of poor and unclear stylized images.

[0225] Optionally, in this embodiment, the step of "performing diffusion processing on the original image according to the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style" may include:

[0226] Based on the style description information, the scene description information, and the facial attribute information, the original image is subjected to diffusion processing to generate an initial stylized image corresponding to the original image under the target image style;

[0227] The initial stylized image is subjected to image quality restoration processing to obtain the target image corresponding to the original image under the target image style.

[0228] The step "performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate an initial stylized image corresponding to the original image under the target image style" may include:

[0229] Based on the image size ratio of the original image, the resolution of the original image is adjusted to obtain the adjusted original image;

[0230] Based on the style description information, the scene description information, and the facial attribute information, the adjusted original image is subjected to diffusion processing to generate a stylized image corresponding to the original image under the target image style.

[0231] Specifically, the resolution of the original image can be adjusted according to the original image size ratio, so as to keep the aspect ratio of the image unchanged and ensure that the image is not distorted.

[0232] In this embodiment, the resolution of the input original image is first adjusted to an appropriate resolution while maintaining the proportionality. After the initial stylization result is output, the image quality enhancement module selects the corresponding image quality repair / super-resolution model for multi-stage repair, which can effectively improve the quality of stylization without affecting the generated target style.

[0233] In specific scenarios, when it is necessary to control the pose distribution of objects in an image, ControlNet can be introduced. ControlNet is a control and regulation network used for deep learning models, which can enhance the controllability and flexibility of generative models (such as diffusion models). By introducing ControlNet, specific control signals can be applied during the generation process, thereby achieving precise control over the generated results.

[0234] Optionally, in this embodiment, the step of "performing diffusion processing on the original image according to the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style" may include:

[0235] Identify a reference image containing the target's pose;

[0236] Extract the pose feature information of the reference image;

[0237] Based on the style description information, the scene description information, the facial attribute information, and the pose feature information, the original image is subjected to diffusion processing to generate a target image corresponding to the original image under the target image style and the target pose.

[0238] In this context, the reference image of the target pose is equivalent to the applied control signal, used to guide image generation.

[0239] There are various methods for extracting pose feature information from the reference image, and this embodiment does not limit this method. For example, pose feature information can be obtained through Canny edge detection; pose feature information can also be obtained through a pose capture algorithm; or pose feature information can be obtained through a sketch line drawing.

[0240] Optionally, in this embodiment, the step "extracting the pose feature information of the reference image" may include:

[0241] Edge detection processing is performed on the reference image to obtain the edge image corresponding to the reference image;

[0242] The edge image is subjected to feature extraction processing to obtain the pose feature information of the reference image.

[0243] In this embodiment, edge feature extraction can guide the pose distribution of objects in the original image. Specifically, the Canny operator can be used to extract the edge image. The process of edge image extraction using the Canny operator can include: smoothing the reference image with a Gaussian filter to remove noise, calculating the gradient magnitude and direction of the noise-removed image, performing non-maximum suppression on the gradient magnitude after calculating the gradient magnitude and direction, and finally using double threshold detection to determine edge points, thereby obtaining the edge image. The edge image provides the contour or structural information of objects or people in the image. By using its feature information as conditional input, ControlNet can incorporate this feature information into the generated image, gradually guiding the structure and pose of the generated image during the denoising process.

[0244] Optionally, in this embodiment, the step "extracting the pose feature information of the reference image" may include:

[0245] Detect the skeletal key points of the object in the reference image;

[0246] Based on the detected skeletal key points, limb movement recognition processing is performed on the object in the reference image to determine the posture feature information of the reference image according to the limb movement recognition results.

[0247] Among them, actions such as running and jumping can be identified by detecting key skeletal points of a person, and then pose feature information can be captured based on the actions. The pose feature information can control the actions of the person in the generated image.

[0248] Optionally, in this embodiment, the step "extracting the pose feature information of the reference image" may include:

[0249] Detect the skeletal key points of the object in the reference image;

[0250] Based on the coordinate information of the detected skeletal key points, the outline information of the object skeleton is drawn.

[0251] Based on the rendering results, the pose feature information of the reference image is determined.

[0252] Specifically, the coordinate information of the key points of the skeleton can be used to draw the skeleton outline information, obtain the sketch line drawing, and then obtain the posture feature information through the drawn line drawing.

[0253] Specifically, ControlNet supports multi-condition control, enabling simultaneous input of multiple conditions. For example, it can use pose control (such as OpenPose) and edge control (such as Canny edge detection) at the same time to further precisely control the pose and details of the generated image.

[0254] Specifically, the step "based on the style description information, the scene description information, the facial attribute information, and the pose feature information, performing diffusion processing on the original image to generate a target image corresponding to the original image under the target image style and the target pose" may include:

[0255] Based on the style description information, the scene description information, and the facial attribute information, determine the prompt information;

[0256] Based on the prompt information, the original image is downsampled at multiple scales to obtain the target downsampled features;

[0257] The pose feature information is fused with the target downsampled features to obtain the target features;

[0258] The target features are upsampled at multiple scales to obtain the image noise information of the original image;

[0259] Based on the image noise information, the original image is denoised to obtain the target image corresponding to the original image under the target image style and the target pose.

[0260] There are various ways to fuse pose feature information with target downsampled features, such as weighted fusion.

[0261] Specifically, the step "based on the prompt information, perform downsampling processing on the original image at multiple scales to obtain the target downsampled features" may include:

[0262] Extract the semantic features of the prompt information and the image features of the original image respectively;

[0263] Attention feature interaction processing is performed on the semantic feature information and the image feature information to obtain attention features;

[0264] The attention features are downsampled to obtain downsampled features;

[0265] The downsampled features are then identified as new image feature information.

[0266] Return to the step of performing attention feature interaction processing on the semantic feature information and the image feature information to obtain attention features, until the number of iterations meets the preset condition, and obtain the target downsampled features.

[0267] Specifically, the step "upsampling the target features at multiple scales to obtain the image noise information of the original image" may include:

[0268] The target features are upsampled to obtain the upsampled features;

[0269] Based on the prompt information, attention processing is performed on the upsampled features to obtain attention-processed features;

[0270] The attention processing features are used as new target features;

[0271] Return to the step of upsampling the target features to obtain the upsampled features, until the number of iterations meets the preset condition, and obtain the image noise information of the original image.

[0272] Optionally, in this embodiment, the step of "performing diffusion processing on the original image according to the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style" may include:

[0273] Obtain a reference image in the style of the target image, and extract the feature information of the reference image;

[0274] Based on the style description information, the scene description information, and the facial attribute information, determine the prompt information;

[0275] Based on the feature information of the reference image and the prompt information, the original image is subjected to diffusion processing to generate a target image corresponding to the original image under the target image style.

[0276] In this embodiment, the stylistic sense of the generated image can be enhanced by introducing reference images in the style of the target image. This embodiment can provide one or more style reference images and utilize an IPA (IP-Adapter) model for image cues. Specifically, the IP-Adapter can inject the style of the reference images into the generated image while retaining the content of the text cues, thereby achieving high-quality style transfer.

[0277] Among them, IP-Adapter is a high-efficiency and lightweight adapter for implementing image cues for pre-trained text-to-image diffusion models. A lightweight IP-Adapter can achieve performance comparable to or even better than a fine-tuned image cue model. IP-Adapter can not only be generalized to other custom models fine-tuned from the same base model, but can also be used in conjunction with controlled generation tools to achieve controlled generation.

[0278] Specifically, the step "based on the feature information of the reference image and the prompt information, perform diffusion processing on the original image to generate a target image corresponding to the original image under the target image style" may include:

[0279] Attention processing is performed on the feature information of the prompt information and the reference image respectively to obtain the attention features of the prompt information and the attention features of the reference image;

[0280] The attention features of the prompt information and the attention features of the reference image are fused together to obtain the target attention features;

[0281] Based on the target attention features, the original image is denoised to obtain the target image corresponding to the target image style.

[0282] There are various ways to fuse the attention features of the prompt information and the attention features of the reference image, and this embodiment does not limit this. For example, the fusion method can be a weighted operation, etc.

[0283] This application provides a general, multi-stage full-image stylization effect optimization scheme. Specifically, it can be based on a diffusion model to perform noise addition and denoising operations on the image to achieve image stylization. Combined with various controllable plugins, it enables the controllable generation of the target stylized image. Through multi-stage workflow design, the generalization of the stylization workflow generation is greatly improved. The targeted introduction of face modules and post-processing modules enhances the aesthetics and quality of the generated workflow. (Reference) Figure 1c , Figure 1c The image on the right is the result of stylizing the image on the left using the image processing method of this application.

[0284] The framework proposed in this application can support some stylized special effects gameplay on video platforms, as well as some template creation in games, for example, it can support Figure 1c The stylized effects, combined with templates, can guide users to upload an image and instantly transport them to a stylized world. Because of its high interactivity and high-quality output, it can significantly improve user satisfaction and increase daily active users.

[0285] Specifically, refer to Figure 1d This diagram illustrates the model framework of this application. It specifically embeds and integrates numerous modules, including an image and text labeling model, a face module, a multi-stage optimization module for small faces, a limb restoration module, and an image quality enhancement module. This effectively addresses issues in current related technologies such as poor quality, unclear images, image distortion, distorted small faces, and difficulty in preserving character attributes in stylized image generation. Compared to current related technologies, this framework exhibits high generalization and scalability, enabling the generation of high-quality stylized images.

[0286] The face module can be used to identify the age, gender, expression, and hairstyle of objects in an image, thereby obtaining a text description of facial attributes (i.e., facial attribute information in the above embodiments). The image-text labeling model can be used to generate scene text descriptions of the image (specifically, scene description information in the above embodiments). The small face multi-stage optimization module can be used to improve the model's attention to the face region and optimize the generated facial region. The limb restoration module can be used to restore distorted limbs in the image. The image quality enhancement module can be used to improve the resolution of the generated image.

[0287] Specifically, based on Figure 1d The model framework allows users to upload an input image and some stylized prompts (i.e., style descriptions), which are then fed into a diffusion model to achieve style transfer. This stylization result relies on the diffusion model's built-in style sense. Before the style descriptions are input into the diffusion model, they can be preprocessed to obtain processed style descriptions. This preprocessing can include typo correction, text cleaning, and word segmentation.

[0288] in, Figure 1dThe model framework introduces a feature refinement strategy to better integrate the target image style with the features of the original image (which, in specific scenarios, could be real-person features). Specifically, a text-image tagging model can be used to obtain scene description information from the original image, highlighting the features that need to be retained. Furthermore, during customized stylization fine-tuning, a LoRa model for character A is trained. Training data for character A is collected, and the training images are tagged. Tagging can emphasize desired character IP features, such as clothing and facial features. During inference, the retention strength of character A's IP features can be controlled by using the tagged feature cues and adjusting the LoRa fusion strength. If the original image is a real-person image, the character stylization model for character A is used to stylize the real-person image. During diffusion model sampling, some real-person features can be retained for fusion, achieving a perfect fusion of the real-person style and the target image style, thus solving the problems of overfitting character features and feature selection.

[0289] Figure 1d In the provided model framework, in response to the current characteristics of poor quality and unclear stylized images, a dynamic resolution adjustment method is set up to maintain the proportion. First, the resolution of the input original image is adjusted to an appropriate resolution while maintaining the proportion. After the stylized result is output, it goes through the image quality enhancement module, which selects the corresponding image quality repair / super-resolution model for multi-stage repair. This can effectively improve the quality of stylization without affecting the generation of the target style.

[0290] Furthermore, to address the issue of lost character attributes in the target style results, the model framework provided in this application introduces a face module (specifically, the face recognition model in the above embodiments). The face module primarily outputs facial-related information such as age, gender, and facial expression features. Depending on the type of information, different facial attribute descriptions are added to the text prompts based on the character attributes to guide the generation results of the diffusion model; this ensures the preservation of the target style's attributes. For example, regarding the gender attribute, if the input original image is female, but the target image style corresponds to a male character, the generated result will be as follows: Figure 1e As shown in the middle image, there are many male characteristics, but when gender cues are introduced, female characteristics are well preserved, making the image more relevant to the user's gender and thus optimizing the overall effect. For example... Figure 1e As shown, from left to right are the original image, the un-optimized stylized image, and the optimized stylized image.

[0291] Furthermore, regarding the issue of small faces, because faces are small and occupy a relatively small proportion of the image, the network's attention to these faces is weaker during inference, resulting in generated results that may appear to have illogical facial feature distribution and poor quality. To address this issue, this application introduces a multi-stage face effect optimization strategy. Specifically, for faces whose face area in the image is less than a preset proportion (e.g., 20%), they are cropped out and redrawn. This ensures that the network's attention is entirely focused on this face area during the redrawing process. The redrawn result is then pasted back into the original image. During the facial region redrawing process, to ensure that the redrawn result blends seamlessly with the original image boundaries, an inpaint image restoration model is introduced, combined with a mask that includes edge transitions for fusion, resulting in a harmonious image. A comparison illustration is shown below. Figure 1f As shown, the results from left to right are before and after restoration. Specifically, this image restoration model can be a diffusion model.

[0292] To address the issue of distorted hands and limbs, this application can use the text inversion method to mitigate it to some extent. Additionally, a hand restoration module can be introduced to further improve the image generation quality. The processing procedure of the hand restoration module can refer to the processing flow of the small face restoration module and the above embodiments, and will not be elaborated here.

[0293] in addition, Figure 1d The dashed box is optional. Specifically, if a style not supported by the original base model is desired, customization is required. This can be achieved by introducing a LoRa mini-model, collecting a small amount of stylized data, and fine-tuning the model to achieve a customized style effect. Furthermore, if the user's pose distribution needs to be controlled, a ControlNet can be introduced. Edges can be extracted using the Canny operator. ControlNet also supports multiple control conditions being introduced simultaneously (such as edges, pose, depth, sketch, keypoints, etc.). In the case of multi-condition input, each conditional image undergoes a feature extraction step, and then is fused with the generated features of the main branch during each denoising step. In some embodiments, if it is necessary to enhance the stylistic sense of the image, one or more reference style maps can be provided, and an IPA (IP-Adapter) model can be used for image cues. The specific processing procedures for these three methods can be found in the descriptions of the above embodiments, and will not be repeated here.

[0294] By combining the introduction of the above modules and proposing multi-stage optimization strategies, the generalizability and versatility of the stylized workflow can be greatly improved, as well as the stylized target effect can be significantly enhanced.

[0295] It is understood that in some embodiments, the small face multi-stage optimization module, the limb repair module, and the image quality enhancement module are optional in parallel, and the order of the modules can be adjusted. In other words, the order in which the three steps of face optimization, limb repair, and image quality enhancement are executed, as well as whether they must be executed, are not restricted.

[0296] The solution provided in this application effectively addresses the problems of poor generation quality, image distortion, blurriness, distorted small faces, and difficulty in maintaining character attributes when processing various complex images uploaded by users in classic stylized workflows. In terms of effect, it can significantly improve the generation quality and effect of the target style. In terms of product benefits, the improved effect injects more vitality into the short video community, increases creativity and fun, improves content quality, and promotes dissemination and interaction. Specifically, it can enhance attractiveness, such as visual appeal; stylized effects can make short videos more unique and engaging, thereby attracting more viewers' attention. Moreover, through consistent stylized effects, brands can establish a unique visual identity in short videos, enhancing brand recall. Additionally, it can increase user engagement and interaction; stylized video content is more likely to generate user interest and interaction, such as likes, comments, and shares; unique stylized effects can increase user viewing time and improve video completion rates. On the other hand, beautifully stylized advertisements are more likely to receive positive user reviews and feedback, increasing user favorability towards the brand; users are more willing to share uniquely stylized advertisement content, thereby expanding the brand's reach and influence.

[0297] The image processing method in this application can also enhance content diversity. Stylization effects provide content creators with more creative ways to express themselves, making video content more diverse and interesting. Through stylization effects, creators can customize video styles according to different themes and audiences, meeting the needs of different users.

[0298] In one specific embodiment, reference is made to Figure 1d Model framework diagram and Figure 1g The flowchart of image stylization processing based on the image processing method of this application is as follows:

[0299] 1001. Obtain style description information for the original image and the target image to be processed;

[0300] 1002. Generate scene description information for the original image;

[0301] 1003. Perform facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image;

[0302] 1004. Based on the style description information, the scene description information, and the facial attribute information, perform diffusion processing on the original image to generate an initial stylized image corresponding to the original image under the target image style;

[0303] 1005. Perform facial adjustment processing on the initial stylized image to obtain a stylized image with adjusted face;

[0304] Specifically, the initial stylized image output by the diffusion model can be used as input to the small face multi-stage optimization module. The small face multi-stage optimization module can first perform face detection on the initial stylized image to determine the object's facial region in the initial stylized image, thereby cropping the object's facial region from the initial stylized image to obtain the object's facial image; then, it generates a target mask image with edge transitions, and redraws the object's facial image based on the target mask image, that is, it redraws the region of the object's facial area in the initial stylized image; finally, it pastes the redrawn object's facial image back to its original position in the initial stylized image to obtain the stylized image after facial adjustment.

[0305] 1006. Perform object limb adjustment processing on the stylized image after facial adjustment to obtain a stylized image after limb adjustment;

[0306] Specifically, the stylized facial image output by the multi-stage optimization module for small faces can be used as input to the limb restoration module. The limb restoration module first performs limb detection on the stylized facial image to determine the target limb region in the stylized facial image, thereby cropping the target limb region from the stylized facial image to obtain the limb image. Then, it generates a target mask image with edge transitions and redraws the limb image based on the target mask image, that is, redraws the region of the target limb in the stylized facial image. Finally, it pastes the redrawn limb image back into its original position in the stylized facial image to obtain the stylized limb image.

[0307] 1007. Perform image quality restoration processing on the stylized image after limb adjustment to obtain the target image (i.e., the target style map) corresponding to the original image under the target image style.

[0308] In one specific embodiment, the limb repair module and the small face multi-stage optimization module can be parallel modules, as shown in the reference. Figure 1h and Figure 1i , Figure 1h For the model architecture diagram, Figure 1i The flowchart for image stylization processing is described in detail below:

[0309] 1101. Obtain style description information for the original image and the target image to be processed;

[0310] 1102. Generate scene description information for the original image;

[0311] 1103. Perform facial recognition processing on the objects contained in the original image to obtain the facial attribute information of the objects in the original image;

[0312] 1104. Based on the style description information, the scene description information, and the facial attribute information, perform diffusion processing on the original image to generate an initial stylized image corresponding to the original image under the target image style;

[0313] 1105. The initial stylized image is cropped to obtain the object's face image; the object's face image is then redrawn to obtain the redrawn object's face image.

[0314] Specifically, the object's facial region in the initial stylized image can be identified by face detection, and then the object's facial region can be cropped to obtain the object's facial image; then a target mask image with edge transitions can be generated, and the object's facial image can be redrawn based on the target mask image to obtain the redrawn object's facial image.

[0315] 1106. The initial stylized image is cropped to obtain a limb image; the limb image is then redrawn to obtain a redrawn limb image.

[0316] Specifically, the object limb region of the initial stylized image can be identified through limb detection, and then the object limb region can be cropped to obtain a limb image; then a target mask image with edge transitions can be generated, and the limb image can be redrawn based on the target mask image to obtain a redrawn limb image.

[0317] 1107. Based on the positional information of the object's facial region and limb region in the initial stylized image, the redrawn object facial image, the redrawn limb image, and the initial stylized image are fused together to obtain the adjusted stylized image.

[0318] 1108. Perform image quality restoration processing on the adjusted stylized image to obtain the target image (i.e., the target style map) corresponding to the original image under the target image style.

[0319] It should be noted that the execution order of steps 1105 and 1106 is not limited. They can be performed simultaneously, or step 1106 can be performed before step 1105, or step 1105 can be performed before step 1106.

[0320] As can be seen from the above, this embodiment can obtain style description information of the original image to be processed and the target image style; generate scene description information of the original image; perform facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image; and perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information to generate the target image corresponding to the original image under the target image style.

[0321] This application not only utilizes style description information, but also combines scene description information and facial attribute information of objects in the original image to perform image stylization processing. This allows the generated stylized target image to better integrate the features of the original image, be closer to the original image, retain some important information of the original image, and significantly improve the quality of the generated image.

[0322] This embodiment will be described from the perspective of an image processing model training device, which can be integrated into an electronic device, such as a server or a terminal.

[0323] This application provides an image processing model training method, such as... Figure 2 As shown, the specific process of training this image processing model can be as follows:

[0324] 201. Obtain training data and an image processing model, wherein the training data includes sample images under the sample image style and style description information of the sample image style.

[0325] The image processing model is used to generate stylized images. This model is a neural network model, specifically a diffusion model.

[0326] The style of the sample images can be artistic, ACG, oil painting, abstract, or ink painting, etc., and this embodiment does not limit this.

[0327] 202. Generate scene description information for the sample image.

[0328] The scene description information may include the number of objects in the image, object information, environmental features in the image, scene atmosphere, main colors of the scene, object behavior in the image, and may even include clothing features and physical characteristics of objects in the image. This embodiment does not impose any restrictions on this.

[0329] Specifically, a graph-to-text (Graph-to-Text) model can be used to perform scene recognition on sample images and generate scene description information. This model can include an image encoder and a text generator. The image encoder extracts image feature information from the sample image, and this extracted feature information is used as input to the text generator to guide the text generation process. The text generator can then progressively generate text based on the image feature information and preset text prompts.

[0330] 203. Perform facial recognition processing on the objects contained in the sample image to obtain the facial attribute information of the objects in the sample image.

[0331] Facial attribute information can include age, gender, expression, hairstyle, and facial features. Facial features can include the position and shape of the eyes, nose, and mouth.

[0332] Specifically, facial recognition models can be used to perform facial recognition processing on objects contained in sample images.

[0333] 204. Perform forward diffusion processing on the sample image to obtain the processed sample image.

[0334] The forward diffusion process for sample images specifically involves adding noise to the sample images. The forward diffusion process is a gradual process of adding noise to the data, starting from the original data (such as an image) and going through a series of steps until the data is completely transformed into noise.

[0335] Specifically, the processed sample image can be obtained by adding noise to the sample image multiple times. This noise addition involves randomly perturbing the feature points in the sample image to introduce noise, such as Gaussian noise. Specifically, noise can be added to the sample image step by step in T steps, resulting in a sample image after T noise additions. During these T iterations of forward diffusion, the sample image gradually loses its unique and distinct features. Finally, when T approaches infinity, the noise-added sample image becomes equivalent to an isotropic Gaussian noise distribution (the added noise can specifically be Gaussian noise).

[0336] This noise-adding process is also a forward diffusion process, in which x is diffused through a diffusion process at each time step t. t-1 Convert to x t After T time steps, the completely noisy data x is obtained. T x T That is, after T rounds of noise addition to the sample image, the noise can follow a Gaussian distribution. The denoising process is specifically a reverse diffusion process.

[0337] 205. Using the image processing model, based on the style description information, the scene description information, and the facial attribute information, perform reverse diffusion processing on the processed sample image to obtain a stylized sample image.

[0338] The reverse diffusion process is the inverse operation of the forward diffusion process. Its goal is to gradually remove noise from a pure noise state and eventually recover the original data.

[0339] Specifically, the step "using the image processing model to perform reverse diffusion processing on the processed sample image based on the style description information, the scene description information, and the facial attribute information to obtain a stylized sample image" may include:

[0340] The prompt information is determined based on the style description information, the scene description information, and the facial attribute information;

[0341] Based on the prompt information, the image processing model predicts the noise information of the processed sample image.

[0342] Based on the noise information, the processed sample image is denoised to obtain a stylized sample image.

[0343] In some embodiments, style description information, scene description information, and facial attribute information can be directly concatenated to obtain the prompt information. In other embodiments, style description information, scene description information, and facial attribute information can be filled into the corresponding positions in a preset prompt template to obtain the filled preset prompt information. The filled preset prompt information can then be used as the overall prompt information to guide the noise reduction process.

[0344] Specifically, the denoising process can include multiple steps, such as a T-step denoising process. In each denoising step, the noise contained in the image is predicted by the image processing model. The predicted noise information of the current image is used to gradually remove noise from the features, thereby obtaining a stylized image after T-step denoising.

[0345] Specifically, the step "predicting the noise information of the processed sample image based on the prompt information using the image processing model" may include:

[0346] Using the image processing model, based on the prompt information, the processed sample image is downsampled at multiple scales to obtain target features;

[0347] The image processing model is used to upsample the target features at multiple scales to obtain the noise information of the processed sample image.

[0348] In some embodiments, the step "based on the prompting information, performing downsampling processing on the processed sample image at multiple scales to obtain target features" may include:

[0349] Extract the semantic features of the prompt information and the image features of the processed sample image, respectively.

[0350] Attention feature interaction processing is performed on the semantic feature information and the image feature information to obtain attention features;

[0351] The attention features are downsampled to obtain downsampled features;

[0352] The downsampled features are then identified as new image feature information.

[0353] Return to the step of performing attention feature interaction processing on the semantic feature information and the image feature information to obtain attention features, until the number of iterations meets the preset condition and the target feature is obtained.

[0354] In some embodiments, the step "upsampling the target features at multiple scales to obtain noise information of the processed sample image" may include:

[0355] The target features are upsampled to obtain the upsampled features;

[0356] Based on the prompt information, attention processing is performed on the upsampled features to obtain attention-processed features;

[0357] The attention processing features are used as new target features;

[0358] Return to the step of upsampling the target features to obtain the upsampled features, until the number of iterations meets the preset condition, and obtain the noise information of the processed sample image.

[0359] The upsampling process progressively generates higher-resolution image features. An attention mechanism is integrated throughout both the downsampling and upsampling processes, allowing cue information to be incorporated into the image features. Simultaneously, through the attention mechanism, the model can refer to the cue information at each denoising step, ensuring that the generated image matches the cue information.

[0360] Specifically, the step "based on the noise information, perform denoising processing on the processed sample image to obtain a stylized sample image" may include:

[0361] Based on the noise information, the processed sample image is denoised to obtain a denoised feature map.

[0362] The denoised feature map is used as the new original image;

[0363] Return to the step of predicting the noise information of the processed sample image based on the prompt information, until the number of iterations meets the preset condition, and obtain the stylized sample image.

[0364] 206. Based on the sample image and the stylized sample image, adjust the parameters of the image processing model to obtain the trained image processing model.

[0365] Specifically, the image similarity between the sample image and the stylized sample image can be calculated, and the loss information can be determined based on the image similarity. Higher image similarity results in lower loss information, and vice versa.

[0366] The training process can involve calculating the loss information, then using the backpropagation algorithm to adjust the parameters of the image processing model, optimizing the parameters of the image processing model based on the loss information, so that the loss information is less than a preset value, and thus obtaining the trained image processing model.

[0367] The trained image processing model can then be used to stylize images. One image processing method follows this process:

[0368] 1. The server obtains style description information for the original image and the target image to be processed;

[0369] 2. The server generates scene description information for the original image;

[0370] 3. The server performs facial recognition processing on the objects contained in the original image to obtain the facial attribute information of the objects in the original image;

[0371] 4. The server uses the trained image processing model to perform diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style.

[0372] Optionally, in this embodiment, after the step "adjusting the parameters of the image processing model based on the sample image and the stylized sample image to obtain the trained image processing model", it may further include:

[0373] Acquire training sample data and preset weight adjustment parameters for the target style task. The training sample data includes sample reference images for the target style and style description information for the target style.

[0374] The weight adjustment parameters are fused with the weight parameters of the image processing model to obtain the fused weight parameters;

[0375] The sample reference image is subjected to noise addition processing to obtain a noisy sample reference image;

[0376] Generate reference scene description information for the sample reference image; and perform facial recognition processing on the objects in the sample reference image to obtain reference facial attribute information of the objects in the sample reference image;

[0377] Using the image processing model, based on the fused weight parameters, the noisy sample reference image is denoised using the style description information of the target style, the reference scene description information, and the reference facial attribute information to obtain the denoised sample reference image.

[0378] The weight adjustment parameters are updated based on the sample reference image and the denoised sample reference image to obtain the updated weight adjustment parameters.

[0379] Specifically, the target style task can be a task that is not supported by the original primitive model of the image processing model, that is, the original primitive model of the image processing model cannot convert the image into an image under the target style.

[0380] The fusion of the weight adjustment parameters with the weight parameters of the image processing model can be achieved by adding the weight adjustment parameters to the weight parameters of the image processing model. Specifically, the weight adjustment parameters can be fused with the weight parameters of a specific layer (such as the attention layer) in the image processing model.

[0381] Specifically, the weight adjustment parameters can be updated based on the similarity between the sample reference image and the denoised sample reference image. This optimizes the weight adjustment parameters so that the denoised sample reference image closely approximates the original sample reference image, resulting in the updated weight adjustment parameters. In particular, the similarity between the denoised sample reference image and the original sample reference image can be greater than a preset value, which can be set according to actual conditions.

[0382] In specific scenarios, when the target image style is an image style that is not supported by the original base model of the image processing model, style customization is required. Specifically, weight adjustment parameters corresponding to the target image style can be introduced. These weight adjustment parameters are low-rank matrices, i.e., LoRa small models. Through these low-rank matrices, the weight parameters of the image processing model can be adjusted, thereby affecting the generation of the image.

[0383] The Lora (Low-Rank Adaptation) model is a parameter tuning method for deep learning models. It adapts the model to new tasks or data by adding a low-rank matrix to the pre-trained model, thereby achieving rapid model adaptation and efficient training.

[0384] Optionally, in this embodiment, after the step "adjusting the parameters of the image processing model based on the sample image and the stylized sample image to obtain the trained image processing model", it may further include:

[0385] Obtain a reference image corresponding to preset placeholder information and content description information of the reference image, wherein the content description information includes the preset placeholder information;

[0386] Set the preset embedded feature corresponding to the preset placeholder identifier information;

[0387] The preset embedding features of the preset placeholder identifier information are fused with the content description information to obtain the processed content description information;

[0388] The reference image is subjected to noise processing to obtain a noisy reference image;

[0389] The processed content description information is subjected to semantic feature extraction to obtain content semantic features;

[0390] Based on the semantic features of the content, the noisy reference image is subjected to image denoising processing to obtain a denoised reference image;

[0391] Based on the denoised reference image and the reference image, the preset embedding features are updated to obtain the embedding features corresponding to the preset placeholder identifier information;

[0392] Add the mapping relationship information between the preset placeholder identifier information and its embedded features to the embedded feature mapping set.

[0393] The preset placeholder information is specifically a placeholder identifier, which can be a text unit or a pseudo-word.

[0394] Among them, the preset embedded features are fused with the content description information. Specifically, the preset embedded features can replace the corresponding preset placeholder information in the content description information.

[0395] This process involves calculating the loss value between the denoised reference image and the target image, and then updating the embedding features of the placeholder information through backpropagation based on the loss value to gradually reduce the difference between the generated image and the target image.

[0396] In this embodiment, a set of images related to placeholder identifiers can be acquired first. Based on these images, a model is trained to learn an embedding feature for the placeholder identifiers. Through an optimization process, the embedding feature of the placeholder identifiers is continuously adjusted so that it can accurately represent the corresponding text units in the latent space. This allows the model to generate images that conform to the target concept based on the embedding feature. For example, if the goal is to reduce limb distortion, the embedding feature of relevant pseudowords can be trained by providing images with natural postures. This allows the model to generate images that are more inclined to natural limb states based on the trained embedding feature.

[0397] Specifically, during online inference using placeholder information and its embedded features, the detail representation of the generated image can be further optimized by adjusting the weights of the embedded features. For example, if distortion is found in the generated image of limbs, the weights of the embedded features related to limbs can be increased to make the model pay more attention to the natural appearance of the limbs.

[0398] As can be seen from the above, this embodiment can acquire training data and an image processing model. The training data includes sample images under the sample image style and style description information of the sample image style; generate scene description information of the sample images; perform facial recognition processing on the objects contained in the sample images to obtain facial attribute information of the objects in the sample images; perform forward diffusion processing on the sample images to obtain processed sample images; through the image processing model, perform reverse diffusion processing on the processed sample images according to the style description information, the scene description information, and the facial attribute information to obtain stylized sample images; and adjust the parameters of the image processing model according to the sample images and the stylized sample images to obtain the trained image processing model.

[0399] This application not only utilizes style description information, but also combines scene description information and facial attribute information of objects in the original image to perform image stylization processing. This allows the generated stylized target image to better integrate the features of the original image, be closer to the original image, retain some important information of the original image, and significantly improve the quality of the generated image.

[0400] To better implement the above methods, this application also provides an image processing and image processing model training system. This image processing and image processing model training system includes an image processing device 31 and an image processing model training device 32. For example... Figure 3a As shown, the image processing device 31 may include a first acquisition unit 3101, a first generation unit 3102, a first recognition unit 3103, and a first diffusion unit 3104; as Figure 3bAs shown, the image processing model training device 32 may include a second acquisition unit 3201, a second generation unit 3202, a second recognition unit 3203, a second diffusion unit 3204, a third diffusion unit 3205, and an adjustment unit 3206, as follows:

[0401] A. Image processing device 31

[0402] (1) First acquisition unit 3101;

[0403] The first acquisition unit is used to acquire style description information of the original image and the target image to be processed.

[0404] (2) First generation unit 3102;

[0405] The first generation unit is used to generate scene description information for the original image.

[0406] (3) First identification unit 3103;

[0407] The first recognition unit is used to perform facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image.

[0408] (4) First diffusion unit 3104;

[0409] The first diffusion unit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate a target image corresponding to the original image under the target image style.

[0410] Optionally, in some embodiments of this application, the first diffusion unit may include a diffusion subunit and a face adjustment subunit, as follows:

[0411] The diffusion subunit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate an initial stylized image corresponding to the original image under the target image style;

[0412] The face adjustment subunit is used to perform object face adjustment processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0413] Optionally, in some embodiments of this application, the face adjustment subunit may be specifically used to crop the object's face region of the initial stylized image to obtain an object's face image; to redraw the object's face image to obtain a redrawn object's face image; and to fuse the redrawn object's face image with the initial stylized image based on the position information of the object's face region in the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0414] Optionally, in some embodiments of this application, the step "redrawing the object's facial image to obtain a redrawn object's facial image" may include:

[0415] Based on the initial stylized image and the positional information of the object's facial region in the initial stylized image, an initial mask image is generated, the initial mask image including masked regions and non-masked regions;

[0416] The edges between the masked and non-masked regions in the initial mask image are blurred to obtain the target mask image;

[0417] Based on the target mask image, the object's facial image is subjected to diffusion processing to obtain a redrawn object's facial image.

[0418] Optionally, in some embodiments of this application, the first diffusion unit may include a diffusion subunit and a limb adjustment subunit, as follows:

[0419] The diffusion subunit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate an initial stylized image corresponding to the original image under the target image style;

[0420] The limb adjustment subunit is used to perform object limb adjustment processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0421] Optionally, in some embodiments of this application, the limb adjustment subunit may be specifically used to perform cropping processing on the object limb region of the initial stylized image to obtain a limb image; to perform redrawing processing on the limb image to obtain a redrawn limb image; and to perform fusion processing on the redrawn limb image and the initial stylized image based on the position information of the object limb region in the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0422] Optionally, in some embodiments of this application, the step "redrawing the limb image to obtain a redrawn limb image" may include:

[0423] Based on the initial stylized image and the positional information of the object's limb region in the initial stylized image, an initial mask image is generated, the initial mask image including masked regions and non-masked regions;

[0424] The edges between the masked and non-masked regions in the initial mask image are blurred to obtain the target mask image;

[0425] Based on the target mask image, the limb image is subjected to diffusion processing to obtain the redrawn limb image.

[0426] Optionally, in some embodiments of this application, the first diffusion unit may include a prompting determination subunit, a noise prediction subunit, and a denoising subunit, as follows:

[0427] The prompt determination subunit is used to determine prompt information based on the style description information, the scene description information, and the facial attribute information;

[0428] A noise prediction subunit is used to predict image noise information of the original image based on the prompt information;

[0429] The denoising subunit is used to perform denoising processing on the original image based on the image noise information to obtain the target image corresponding to the original image under the target image style.

[0430] Optionally, in some embodiments of this application, the noise prediction subunit may be specifically used to: when a preset text unit is identified from the prompt information, search for the target embedding feature corresponding to the preset text unit from a feature mapping set, the feature mapping set including mapping relationship information between text units and embedding features; fuse the target embedding feature with the prompt information to obtain processed prompt information; perform semantic feature extraction processing on the processed prompt information to obtain prompt semantic features; and predict the image noise information of the original image based on the prompt semantic features.

[0431] Optionally, in some embodiments of this application, the noise prediction subunit may be used to obtain the weight adjustment parameters corresponding to the style description information; fuse the weight adjustment parameters with the weight parameters of the image processing model to obtain the fused weight parameters; and, based on the fused weight parameters, perform attention feature interaction processing on the prompt information and the original image through the image processing model to predict the image noise information of the original image.

[0432] Optionally, in some embodiments of this application, the step "fusing the weight adjustment parameters with the weight parameters of the image processing model to obtain fused weight parameters" may include:

[0433] Based on the feature retention intensity information of the original image, the fusion intensity information of the weight adjustment parameters is determined;

[0434] Based on the fusion intensity information, the weight adjustment parameters are fused with the weight parameters of the image processing model to obtain the fused weight parameters.

[0435] Optionally, in some embodiments of this application, the first diffusion unit may include a diffusion subunit and an image quality restoration subunit, as follows:

[0436] The diffusion subunit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate an initial stylized image corresponding to the original image under the target image style;

[0437] The image quality restoration subunit is used to perform image quality restoration processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style.

[0438] Optionally, in some embodiments of this application, the first diffusion unit may include a determination subunit, a feature extraction subunit, and a first diffusion processing subunit, as follows:

[0439] The determining subunit is used to determine a reference image containing the target pose;

[0440] A feature extraction subunit is used to extract pose feature information of the reference image;

[0441] The first diffusion processing subunit is used to perform diffusion processing on the original image based on the style description information, the scene description information, the facial attribute information and the pose feature information, so as to generate a target image corresponding to the original image under the target image style and the target pose.

[0442] Optionally, in some embodiments of this application, the first diffusion unit may include an image acquisition subunit, a prompt information determination subunit, and a second diffusion processing subunit, as follows:

[0443] The image acquisition subunit is used to acquire a reference image under the target image style and extract feature information from the reference image;

[0444] The prompt information determination subunit is used to determine the prompt information based on the style description information, the scene description information, and the facial attribute information;

[0445] The second diffusion processing subunit is used to perform diffusion processing on the original image based on the feature information of the reference image and the prompt information, so as to generate a target image corresponding to the original image under the target image style.

[0446] B. Image processing model training device 32;

[0447] (5) Second acquisition unit 3201;

[0448] The second acquisition unit is used to acquire training data and an image processing model. The training data includes sample images under the sample image style and style description information of the sample image style.

[0449] (6) Second generation unit 3202;

[0450] The second generation unit is used to generate scene description information for the sample image.

[0451] (7) Second identification unit 3203;

[0452] The second recognition unit is used to perform facial recognition processing on the objects contained in the sample image to obtain facial attribute information of the objects in the sample image.

[0453] (8) Second diffusion unit 3204;

[0454] The second diffusion unit is used to perform forward diffusion processing on the sample image to obtain a processed sample image.

[0455] (9) Third diffusion unit 3205;

[0456] The third diffusion unit is used to perform reverse diffusion processing on the processed sample image based on the style description information, the scene description information, and the facial attribute information using the image processing model, to obtain a stylized sample image.

[0457] (10) Adjustment unit 3206;

[0458] The adjustment unit is used to adjust the parameters of the image processing model based on the sample image and the stylized sample image to obtain the trained image processing model.

[0459] As can be seen from the above, in this embodiment, the first acquisition unit 3101 acquires style description information of the original image to be processed and the target image style; the first generation unit 3102 generates scene description information of the original image; the first recognition unit 3103 performs facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image; and the first diffusion unit 3104 performs diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information to generate the target image corresponding to the original image under the target image style.

[0460] Alternatively, the second acquisition unit 3201 acquires training data and an image processing model, the training data including sample images under the sample image style and style description information of the sample image style; the second generation unit 3202 generates scene description information of the sample images; the second recognition unit 3203 performs facial recognition processing on the objects contained in the sample images to obtain facial attribute information of the objects in the sample images; the second diffusion unit 3204 performs forward diffusion processing on the sample images to obtain processed sample images; the third diffusion unit 3205 performs reverse diffusion processing on the processed sample images using the image processing model, based on the style description information, the scene description information, and the facial attribute information, to obtain stylized sample images; and the adjustment unit 3206 adjusts the parameters of the image processing model based on the sample images and the stylized sample images to obtain the trained image processing model.

[0461] This application can not only utilize style description information, but also combine scene description information and facial attribute information of objects in the original image to perform image stylization processing. This allows the generated stylized target image to better integrate the features of the original image, be closer to the original image, retain some important information of the original image, and greatly improve the quality of the generated image.

[0462] This application also provides an electronic device, such as... Figure 4 The diagram shows a structural schematic of an electronic device involved in an embodiment of this application. This electronic device can be a terminal or a server, specifically:

[0463] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0464] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0465] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0466] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0467] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0468] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0469] The process involves: acquiring style description information for the original image and the target image style; generating scene description information for the original image; performing facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image; and performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style.

[0470] Alternatively, training data and an image processing model can be obtained. The training data includes sample images in a sample image style and style description information of the sample image style. Scene description information of the sample images is generated. Facial recognition processing is performed on the objects contained in the sample images to obtain facial attribute information of the objects in the sample images. Forward diffusion processing is performed on the sample images to obtain processed sample images. Through the image processing model, based on the style description information, the scene description information, and the facial attribute information, backward diffusion processing is performed on the processed sample images to obtain stylized sample images. The parameters of the image processing model are adjusted based on the sample images and the stylized sample images to obtain the trained image processing model.

[0471] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0472] As can be seen from the above, this embodiment can obtain style description information of the original image to be processed and the target image style; generate scene description information of the original image; perform facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image; and perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information to generate the target image corresponding to the original image under the target image style.

[0473] Alternatively, training data and an image processing model can be obtained. The training data includes sample images in a sample image style and style description information of the sample image style. Scene description information of the sample images is generated. Facial recognition processing is performed on the objects contained in the sample images to obtain facial attribute information of the objects in the sample images. Forward diffusion processing is performed on the sample images to obtain processed sample images. Through the image processing model, based on the style description information, the scene description information, and the facial attribute information, backward diffusion processing is performed on the processed sample images to obtain stylized sample images. The parameters of the image processing model are adjusted based on the sample images and the stylized sample images to obtain the trained image processing model.

[0474] This application can not only utilize style description information, but also combine scene description information and facial attribute information of objects in the original image to perform image stylization processing. This allows the generated stylized target image to better integrate the features of the original image, be closer to the original image, retain some important information of the original image, and greatly improve the quality of the generated image.

[0475] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0476] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image processing methods and image processing model training methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0477] The process involves: acquiring style description information for the original image and the target image style; generating scene description information for the original image; performing facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image; and performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style.

[0478] Alternatively, training data and an image processing model can be obtained. The training data includes sample images in a sample image style and style description information of the sample image style. Scene description information of the sample images is generated. Facial recognition processing is performed on the objects contained in the sample images to obtain facial attribute information of the objects in the sample images. Forward diffusion processing is performed on the sample images to obtain processed sample images. Through the image processing model, based on the style description information, the scene description information, and the facial attribute information, backward diffusion processing is performed on the processed sample images to obtain stylized sample images. The parameters of the image processing model are adjusted based on the sample images and the stylized sample images to obtain the trained image processing model.

[0479] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0480] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0481] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the image processing methods and image processing model training methods provided in the embodiments of this application, the beneficial effects that any of the image processing methods and image processing model training methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0482] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations of the image processing described above.

[0483] The foregoing has provided a detailed description of an image processing method, an image processing model training method, and related equipment provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: Obtain style description information for the original and target images to be processed; The scene description information for generating the original image; Facial recognition processing is performed on the objects contained in the original image to obtain the facial attribute information of the objects in the original image; Based on the style description information, the scene description information, and the facial attribute information, the original image is subjected to diffusion processing to generate a target image corresponding to the original image under the target image style.

2. The method according to claim 1, characterized in that, The step of performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style includes: Based on the style description information, the scene description information, and the facial attribute information, the original image is subjected to diffusion processing to generate an initial stylized image corresponding to the original image under the target image style; The initial stylized image is subjected to object facial adjustment processing to obtain the target image corresponding to the original image under the target image style.

3. The method according to claim 2, characterized in that, The step of performing facial adjustment processing on the initial stylized image to obtain the target image corresponding to the original image under the target image style includes: The initial stylized image is cropped to obtain the object's facial image; The object's facial image is redrawn to obtain a redrawn object's facial image. Based on the positional information of the object's facial region in the initial stylized image, the redrawn object's facial image is fused with the initial stylized image to obtain the target image corresponding to the original image under the target image style.

4. The method according to claim 3, characterized in that, The step of redrawing the object's facial image to obtain a redrawn object's facial image includes: Based on the initial stylized image and the positional information of the object's facial region in the initial stylized image, an initial mask image is generated, the initial mask image including masked regions and non-masked regions; The edges between the masked and non-masked regions in the initial mask image are blurred to obtain the target mask image; Based on the target mask image, the object's facial image is subjected to diffusion processing to obtain a redrawn object's facial image.

5. The method according to claim 1, characterized in that, The step of performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style includes: Based on the style description information, the scene description information, and the facial attribute information, the original image is subjected to diffusion processing to generate an initial stylized image corresponding to the original image under the target image style; The initial stylized image is subjected to object limb adjustment processing to obtain the target image corresponding to the original image under the target image style.

6. The method according to claim 5, characterized in that, The step of adjusting the object's limbs in the initial stylized image to obtain the target image corresponding to the original image under the target image style includes: The initial stylized image is cropped to obtain a limb image; The limb image is redrawn to obtain a redrawn limb image; Based on the positional information of the object's limb region in the initial stylized image, the redrawn limb image is fused with the initial stylized image to obtain the target image corresponding to the original image under the target image style.

7. The method according to claim 6, characterized in that, The step of redrawing the limb image to obtain a redrawn limb image includes: Based on the initial stylized image and the positional information of the object's limb region in the initial stylized image, an initial mask image is generated, the initial mask image including masked regions and non-masked regions; The edges between the masked and non-masked regions in the initial mask image are blurred to obtain the target mask image; Based on the target mask image, the limb image is subjected to diffusion processing to obtain the redrawn limb image.

8. The method according to claim 1, characterized in that, The step of performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style includes: The prompt information is determined based on the style description information, the scene description information, and the facial attribute information; Based on the prompt information, predict the image noise information of the original image; Based on the image noise information, the original image is denoised to obtain the target image corresponding to the original image under the target image style.

9. The method according to claim 8, characterized in that, The step of predicting image noise information of the original image based on the prompt information includes: When a preset text unit is identified from the prompt information, the target embedding feature corresponding to the preset text unit is searched from the feature mapping set, which includes the mapping relationship information between the text unit and the embedding feature; The target embedded features are fused with the prompt information to obtain the processed prompt information; The processed prompt information is subjected to semantic feature extraction to obtain prompt semantic features; Based on the semantic features of the prompts, predict the image noise information of the original image.

10. The method according to claim 8, characterized in that, The step of predicting image noise information of the original image based on the prompt information includes: Obtain the weight adjustment parameters corresponding to the style description information; The weight adjustment parameters are fused with the weight parameters of the image processing model to obtain the fused weight parameters. Using the image processing model, based on the fused weight parameters, attention feature interaction processing is performed on the prompt information and the original image to predict the image noise information of the original image.

11. The method according to claim 10, characterized in that, The step of fusing the weight adjustment parameters with the weight parameters of the image processing model to obtain the fused weight parameters includes: Based on the feature retention intensity information of the original image, the fusion intensity information of the weight adjustment parameters is determined; Based on the fusion intensity information, the weight adjustment parameters are fused with the weight parameters of the image processing model to obtain the fused weight parameters.

12. The method according to claim 1, characterized in that, The step of performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style includes: Based on the style description information, the scene description information, and the facial attribute information, the original image is subjected to diffusion processing to generate an initial stylized image corresponding to the original image under the target image style; The initial stylized image is subjected to image quality restoration processing to obtain the target image corresponding to the original image under the target image style.

13. The method according to claim 1, characterized in that, The step of performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style includes: Identify a reference image containing the target's pose; Extract the pose feature information of the reference image; Based on the style description information, the scene description information, the facial attribute information, and the pose feature information, the original image is subjected to diffusion processing to generate a target image corresponding to the original image under the target image style and the target pose.

14. The method according to claim 1, characterized in that, The step of performing diffusion processing on the original image based on the style description information, the scene description information, and the facial attribute information to generate a target image corresponding to the original image under the target image style includes: Obtain a reference image in the style of the target image, and extract the feature information of the reference image; Based on the style description information, the scene description information, and the facial attribute information, determine the prompt information; Based on the feature information of the reference image and the prompt information, the original image is subjected to diffusion processing to generate a target image corresponding to the original image under the target image style.

15. A method for training an image processing model, characterized in that, include: Acquire training data and an image processing model, wherein the training data includes sample images under the sample image style and style description information of the sample image style; The scene description information of the sample image is generated; Facial recognition processing is performed on the objects contained in the sample image to obtain the facial attribute information of the objects in the sample image; The sample image is subjected to forward diffusion processing to obtain the processed sample image; Using the image processing model, the processed sample image is subjected to reverse diffusion processing based on the style description information, the scene description information, and the facial attribute information to obtain a stylized sample image. Based on the sample image and the stylized sample image, the parameters of the image processing model are adjusted to obtain the trained image processing model.

16. An image processing apparatus, characterized in that, include: The first acquisition unit is used to acquire style description information of the original image to be processed and the target image style; The first generation unit is used to generate scene description information of the original image; The first recognition unit is used to perform facial recognition processing on the objects contained in the original image to obtain facial attribute information of the objects in the original image. The first diffusion unit is used to perform diffusion processing on the original image according to the style description information, the scene description information and the facial attribute information, so as to generate a target image corresponding to the original image under the target image style.

17. An image processing model training device, characterized in that, include: The second acquisition unit is used to acquire training data and an image processing model. The training data includes sample images under the sample image style and style description information of the sample image style. The second generation unit is used to generate scene description information for the sample image; The second recognition unit is used to perform facial recognition processing on the objects contained in the sample image to obtain facial attribute information of the objects in the sample image. The second diffusion unit is used to perform forward diffusion processing on the sample image to obtain a processed sample image. The third diffusion unit is used to perform reverse diffusion processing on the processed sample image based on the style description information, the scene description information and the facial attribute information through the image processing model to obtain the stylized sample image; The adjustment unit is used to adjust the parameters of the image processing model based on the sample image and the stylized sample image to obtain the trained image processing model.

18. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program in the memory to perform the operations in the image processing method of any one of claims 1 to 14 or the image processing model training method of claim 15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform steps in the image processing method of any one of claims 1 to 14 or the image processing model training method of claim 15.

20. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the image processing method according to any one of claims 1 to 14 or the image processing model training method according to claim 15.

Images