Image processing method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses an image processing method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a customized target IP image picture, carrying out the low-rank fine tuning training of a first diffusion model through employing the target IP image picture, and obtaining a reference IP image generation model, performing training material generation based on the target IP image picture and the reference IP image generation model to obtain at least one IP image generation training material, and performing second diffusion model low-rank fine tuning training on the reference IP image generation model based on the IP image generation training material and the target IP image to obtain a target IP image generation model.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an image processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] In today's digital age, virtual IP characters have become an important element in brand marketing, the entertainment industry, and online interaction. From brand virtual IPs and digital IPs to game characters, a unified, stable, and attractive IP character is a key digital asset for building user awareness and conveying brand value. These virtual IP characters need to be applied to various visual media such as posters, advertisements, and social media content to create a unique and charming IP character that fits the brand's image. This plays an irreplaceable role in strengthening brand awareness, enhancing brand influence, and expanding brand market boundaries. Summary of the Invention

[0003] This specification provides an image processing method, apparatus, storage medium, and electronic device, the technical solutions of which are as follows:

[0004] Firstly, embodiments of this specification provide an image processing method, the method comprising:

[0005] Obtain a customized target IP image, and use the target IP image to perform low-rank fine-tuning training of the first diffusion model to obtain a reference IP image generation model;

[0006] At least one IP image generation training material is obtained by generating training materials based on the target IP image and the reference IP image generation model.

[0007] Based on the IP image generation training materials and the target IP image, the reference IP image generation model is trained by performing low-rank fine-tuning of the second diffusion model to obtain the target IP image generation model.

[0008] Secondly, embodiments of this specification provide an image processing device, the device comprising:

[0009] The material acquisition module is used to acquire customized target IP image images, and to use the target IP image images to perform low-rank fine-tuning training of the first diffusion model to obtain a reference IP image generation model.

[0010] The material acquisition module is used to generate training materials for at least one IP image generation training material for the reference IP image generation model by employing at least one image and scene decoupling generation strategy.

[0011] The model fine-tuning module is used to perform low-rank fine-tuning training of the reference IP image generation model based on the IP image generation training materials and the target IP image image to obtain the target IP image generation model.

[0012] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0013] Fourthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0014] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0015] In one or more embodiments of this specification, the electronic device uses a customized target IP image to perform low-rank fine-tuning training of a first diffusion model to obtain a reference IP image generation model. Based on the target IP image and the reference IP image generation model, training materials are generated to obtain at least one IP image generation training material. Based on the IP image generation training material and the target IP image, the reference IP image generation model is subjected to low-rank fine-tuning training of a second diffusion model to obtain the target IP image generation model. Compared to the high-cost traditional model that relies on continuous investment from designers or 3D modeling teams, this method provides an AI generation tool that can be reused infinitely with a one-time model training investment. This significantly reduces the customization costs for IP character visual materials. Furthermore, the target IP character image generation model can quickly generate images of specified target IP characters based on text prompts, greatly shortening the content creation cycle. This allows IP characters to keep pace with social trends and efficiently integrate with various promotional activities, enhancing their influence. The resulting "target model" possesses both extremely high IP character fidelity and rich scene generation capabilities. The target IP character image generation model can stably generate IP characters that are highly consistent with expectations, significantly improving the accuracy and stability of the output images and reducing the number of times users need to repeatedly generate and filter images. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of this specification;

[0018] Figure 2 This is a flowchart illustrating a low-rank fine-tuning training method provided in the embodiments of this specification;

[0019] Figure 3 This is a schematic diagram of a target IP image provided in the embodiments of this specification;

[0020] Figure 4 This is a schematic diagram illustrating the process of generating a first training material image provided in an embodiment of this specification;

[0021] Figure 5 This is a schematic diagram of a scene for generating a first training material image provided in an embodiment of this specification;

[0022] Figure 6 This is a schematic diagram illustrating the process of generating a second training material image according to an embodiment of this specification;

[0023] Figure 6a , Figure 6b as well as Figure 6c It is a set of layered control image generation and processing effect diagrams;

[0024] Figure 7 This is a schematic diagram of a process for generating a third training material image provided in the embodiments of this specification;

[0025] Figure 8 This is a schematic diagram of a region division scenario provided in the embodiments of this specification;

[0026] Figure 9 This is a comparison diagram of the generation effect of partition control provided in the embodiments of this specification;

[0027] Figure 10 This is a schematic diagram of the structure of an image processing device provided in the embodiments of this specification;

[0028] Figure 11 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0029] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0030] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0031] In related technologies, with the widespread application of digital technology in brand promotion scenarios, creating IP character models that conform to brand characteristics and have unique recognizability has become an important requirement. In practical applications, IP character customization often uses design teams, but this approach has exposed many shortcomings that urgently need to be addressed:

[0032] Traditional IP digital characters require designers to create comics or 3D models and renders, which is expensive. Moreover, each time these characters need to be used in different contexts, that is, when the IP character needs images in different contexts, the relevant team needs to output IP character images for the corresponding context. This will generate continuous costs, which undoubtedly increases the cost burden. The manual approach also limits the flexibility and autonomy of brand image creation.

[0033] The present specification will now be described in detail with reference to specific embodiments.

[0034] In one embodiment, such as Figure 1As shown, a visual processing method is proposed, which can be implemented using a computer program and can run on a visual processing device based on the von Neumann architecture. This computer program can be integrated into an application or run as a standalone utility application. The visual processing device can be an electronic device, including but not limited to: personal computers, tablets, handheld devices, server devices, wearable devices, computing devices, or other processing devices connected to a wireless modem. In different networks, terminal devices can be called by different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user equipment, cellular phone, cordless phone, terminal device in 5G networks or future evolved networks, etc.

[0035] Specifically, the image processing method includes:

[0036] S102: Obtain a customized target IP image image, and use the target IP image image to perform low-rank fine-tuning training of the first diffusion model to obtain a reference IP image generation model;

[0037] Target IP Image: Refers to a set or more digital images that clearly and standardly display the core design features of a specific intellectual property (IP) image, forming the initial training set. Target IP image images can be manually designed, or they can be generated using a text-based multimodal model, where users customize IP image description prompts, and the images are output through drawing processing using the text-based multimodal model.

[0038] In an optional embodiment of the present invention, the background of the image is transparent or a solid color background to facilitate the model's learning of the inherent features of the image.

[0039] The first diffusion model low-rank fine-tuning training refers to the process of efficiently fine-tuning the initial parameters of a general, pre-trained basic diffusion multimodal large model using target IP image images.

[0040] Optionally, the basic diffusion multimodal large model can be the Tongyi Qianwen series large model, the Hunyuan series large model, the GPT series large model, etc., and there is no limitation here;

[0041] Optionally, in some embodiments, the core of this training is to update only a small number of newly added low-rank matrix parameters injected into specific layers of the base model (such as U-Net network layers), while most of the original parameters of the base model remain frozen.

[0042] Reference IP Image Generation Model: This refers to the initial model obtained after low-rank fine-tuning training of the first diffusion model. The key technical feature of this model is "overfitting," meaning it has extremely high fidelity in reproducing the target IP image, but poor generalization ability in areas such as scene generation.

[0043] In one feasible implementation, the electronic device acquires a preset number of target IP image images. Each acquired image is preprocessed to ensure its background is transparent or solid color, eliminating background information interference with subsequent model learning. Next, the preprocessed image set is annotated with text. This process includes two levels: first, adding descriptive tags to each image based on its content (e.g., posture, action, expression); second, uniformly annotating the entire image set with preset, unique IP image trigger words. Then, low-rank fine-tuning training of the first diffusion model is initiated. This training uses the annotated target IP image images as input, and pre-trained basic diffusion multimodal large model is inserted into an IP image processing plugin layer for the IP image processing scenario to obtain the first diffusion model. The IP image processing plugin layer is created based on machine learning models (such as deep learning models) in related technologies.

[0044] The first diffusion model is trained on a general basic diffusion multimodal large model using labeled target IP image as input. During the training iteration, the first diffusion model receives noisy training samples (which can be training samples after adding noise to the target IP image) and corresponding text annotations, performs denoising image prediction, and outputs the model prediction results in each round of training.

[0045] Then, based on the difference between the model's predicted results and the original clear target IP image, the model processing loss (Loss) is calculated using pre-screening model loss formulas (such as Euclidean distance loss, hinge loss, and cross-entropy loss). Based on this model processing loss, the gradient is calculated using the backpropagation algorithm, and only the parameters of the low-rank matrix (i.e., the "IP image processing plugin layer") injected into the first diffusion model are fine-tuned and updated. The original parameters of the basic diffusion multimodal large model are not updated during this process.

[0046] After the training converges or reaches the preset number of training rounds, the trained low-rank matrix parameters are saved. This parameter file, together with the basic diffusion model, forms the "reference IP image generation model".

[0047] S104: Based on the target IP image and the reference IP image generation model, at least one IP image generation training material is generated;

[0048] Training material generation: Considering the overfitting problem of the reference IP image generation model generated in step S102, a preset technical strategy is adopted to create new and diverse training images through synthesis.

[0049] In some embodiments, the preset technical strategy can be a strategy to decouple the image from the scene. The reference IP image generation model directly trained in step S102 may experience overfitting. In practical applications, it is expected that the background elements of the IP image generated by the large model are simple and not rich, resulting in poor generalization. The background of the generated image is very simple and monotonous, and only some simple images can be produced. However, the image fidelity of the IP image is relatively high. Based on this, the preset technical strategy is used to decouple the image from the scene. The training set generated by this strategy is used to retrain the original large model. The trained model enables the generated data to simultaneously satisfy the two characteristics of "maintaining the fidelity of the IP image" and "introducing background diversity". In this way, the IP image generation training material obtained by using this strategy can achieve the effect of decoupling the image from the scene.

[0050] The scene decoupling generation strategy can adopt a procedural decoupling method based on image editing and graph-generated images. First, a general image generation model generates a reference IP image scene image that is not the target IP image based on the text description of the target scene. Then, the target IP image is synthesized into the "reference IP image scene image" to obtain the IP image scene image. Finally, the reference IP image generation model is used as an "identity corrector" to denoise and redraw the IP image scene image. Because the reference IP image generation model has high IP image fidelity, it denoises and redraws the image, so that the synthesized part is accurately and naturally corrected to the correct appearance of the target IP image, thus obtaining the IP image generation training material for the target IP image, thereby ensuring the accuracy of the target IP.

[0051] The scene decoupling generation strategy can adopt a structural decoupling method based on model hierarchical control. In the model structure of the reference IP image generation model, the target IP image prompt is normally input into the reference IP image generation model. During the model processing, the "identity feature layer group" responsible for IP identity features is separated from the network layer responsible for scene / style. When generating the image, the weights of the reference model (i.e., using LoRA method) are activated only on the "identity feature layer group", while the other network layers are not affected. By selectively activating the network layer weights, the generation of the IP image is precisely controlled by the activated model weight parameters trained using LoRA method, while the background, style and composition of the image are freely generated by the unaffected other network layers based on the input text prompt. Thus, decoupling is achieved within the model to realize the parallel and independent control of IP identity and scene description.

[0052] LoRA (Low-Rank Adaptation of Large Language Models), also known as low-rank fine-tuning, is a parameter-efficient low-rank fine-tuning method. In this specification, its core objective is to adapt the model to specific downstream IP image processing tasks by training a very small number of additional parameters without modifying the original parameters of the pre-trained large model (such as a diffusion model or a large language model). A model trained using the low-rank fine-tuning method can be called a LoRA model, and the model weight parameters adjusted during training can be called LoRA weights.

[0053] The scene decoupling generation strategy can employ a spatial decoupling approach based on spatial partitioning control. Specifically, the image generation canvas of the IP image generation model is divided into different regions. The generation of the target IP image is confined to the IP image region, while the other regions serve as scene regions. Independent text prompts are assigned to each region. Specifically, a first prompt containing the IP image trigger word is assigned to the IP image region, and a second prompt describing the background content is assigned to the scene region. During image generation, the control model generates the IP image within the IP image region based on the first prompt word and a reference model, while simultaneously generating independent scene elements within the scene region based on the second prompt word. These elements are ultimately combined into a complete image with controlled composition.

[0054] S106: Based on the IP image generation training material and the target IP image, the reference IP image generation model is trained by low-rank fine-tuning of the second diffusion model to obtain the target IP image generation model.

[0055] The second diffusion model low-rank fine-tuning training refers to the second (or final) stage of low-rank fine-tuning training performed in the iterative process of this embodiment of the invention. This training uses an expanded and enhanced combined training set containing diverse scenarios, with the aim of improving the model's scenario generation capability and generalization.

[0056] Target IP Image Generation Model: This refers to the final, performance-optimized model. This model possesses both high-fidelity generation capabilities for target IP images and high-fidelity generation capabilities across diverse scenarios. It improves upon the overfitting issue of the initial reference model, elevating its ability from accurate "reproduction" of IP images to accurate "creation." It can stably output target IP image images corresponding to different scenarios in various contexts.

[0057] First, the datasets from the two sources are merged to construct a completely new, final training set. Specifically, the initial "target IP image images" (i.e., the first training set) obtained in step S102 are combined with the "IP image generation training materials" (i.e., the second training set) generated in step S104. The merged final training set has a much larger data scale and greater content diversity than the initial first training set.

[0058] Next, the low-rank fine-tuning training of the second diffusion model is initiated. This training uses the final training set obtained in the previous step as input data. The training can continue based on the reference IP image generation model obtained in step S102, or a completely new low-rank fine-tuning training can be started on the general basic diffusion model.

[0059] During the training iterations, the model simultaneously learns high-fidelity image features from the first training set and generalized knowledge of image-scene fusion from the second training set. Through backpropagation and parameter fine-tuning, the model's weights (specifically the LoRA plugin layer, also known as the "IP image processing plugin layer") are continuously optimized, enabling it to respond to complex scene descriptions in text prompts while ensuring the stability of IP image features.

[0060] After training, the model's performance is evaluated. This can be done qualitatively (by manually observing the generated images) or quantitatively (e.g., calculating fidelity). If the model's performance (e.g., fidelity greater than 92%) meets the preset requirements, training is complete, and the resulting model is the target IP image generation model. If the model's performance does not meet the requirements, the process returns to step S104, where one or more decoupled generation strategies are used to generate more or higher-quality training materials. Step S106 is then repeated, forming a closed loop of "generation-training-evaluation" until the model's performance meets the standards.

[0061] The fidelity rate is calculated as follows: Assuming no image corruption errors occur, and N images are generated, each generated image is compared to the IP image in the training set for similarity. If the cosine distance between the generated image and the IP image in the training set is greater than 0.5, the generated image is considered to belong to the same IP image as the training set. The number of images meeting this condition is counted as M out of the N images; the fidelity rate is then M / N. This fidelity rate metric fully demonstrates the excellent performance of the model in this specification in terms of the accuracy and stability of generated images, effectively solving the problem of existing technologies easily deviating from expectations in generated character images, and providing a high-fidelity virtual character image generation solution.

[0062] In the embodiments of this specification, the electronic device uses a customized target IP image to perform low-rank fine-tuning training of a first diffusion model to obtain a reference IP image generation model. Based on the target IP image and the reference IP image generation model, training materials are generated to obtain at least one IP image generation training material. Based on the IP image generation training material and the target IP image, the reference IP image generation model is trained using a second diffusion model low-rank fine-tuning training to obtain the target IP image generation model. Compared to the high-cost traditional model that relies on continuous investment from designers or 3D modeling teams, this method provides an AI generation tool that can be reused infinitely with a one-time model training investment. This significantly reduces the customization costs for IP character visual materials. Furthermore, the target IP character image generation model can quickly generate images of specified target IP characters based on text prompts, greatly shortening the content creation cycle. This allows IP characters to keep pace with social trends and efficiently integrate with various promotional activities, enhancing their influence. The resulting "target model" possesses both extremely high IP character fidelity and rich scene generation capabilities. The target IP character image generation model can stably generate IP characters that are highly consistent with expectations, significantly improving the accuracy and stability of the output images and reducing the number of times users need to repeatedly generate and filter images.

[0063] Optional, please see Figure 2 , Figure 2 This is a flowchart illustrating a low-rank fine-tuning training method proposed in this specification. Specifically, the process of obtaining a customized target IP image and using that target IP image for low-rank fine-tuning training of the first diffusion model to obtain a reference IP image generation model can be carried out in the following manner:

[0064] S202: Obtain a preset number of target IP image images of a preset background film type, and label the target IP image images with IP image trigger words;

[0065] Target IP Image: refers to a set of one or more digital images that can clearly and standardly display the core design features of a specific intellectual property (IP) image. This set of images will be used as the initial training set.

[0066] Preset background image type: This refers to the image background type preset to reduce the learning difficulty of the model and allow it to focus on the inherent characteristics of the IP image itself. In some embodiments, this type is a transparent background or a solid color background (e.g., a white background), such as... Figure 3 As shown, Figure 3 This is an illustration of a target IP character image, using the Goose Little Sky character as an example. Figure 3 The initial data used to train the Goose Little Sky IP character is shown, which consists of images of various poses against a transparent background.

[0067] Preset customized quantity: refers to the number of images used for initial training. In this embodiment of the invention, this quantity is usually small, for example, a quantity threshold is set, and the preset customized quantity is less than this threshold, which is sufficient for the model to learn the core features of the IP image.

[0068] IP Image Trigger Word: This refers to a unique text identifier or keyword assigned to a target IP image. In the subsequent text-to-image generation process, this trigger word is used to precisely and stably activate and invoke a specially trained model to generate that specific IP image.

[0069] In one feasible implementation, the target IP image images of a "preset customized quantity" are first obtained. During the acquisition process, the images are filtered or their backgrounds are processed to ensure that their backgrounds meet the requirements of the "preset background negative type". For example, all PNG format images with transparent backgrounds are filtered or processed.

[0070] Next, text annotations are performed on all the images acquired in this batch. This annotation process may further include two sub-steps:

[0071] Descriptive annotations: Add relevant descriptive words or tags to each image based on its specific content, such as the IP character's posture (e.g., standing, running), actions (e.g., waving, reading), and expressions.

[0072] Trigger word annotation: The preset, unique "IP image trigger words" are uniformly and without omission added to the annotation information of each image in this batch.

[0073] After this step is completed, the output is an initial dataset of processed and labeled target IP image images that can be directly used for model training.

[0074] S204: Construct an initial IP image generation model, which includes a basic multimodal large model and an IP image processing plugin layer for IP image generation scenarios;

[0075] Initial IP Image Generation Model: This refers to a composite model architecture built before parameter fine-tuning training. This model consists of a large pre-trained multimodal model (such as a diffusion model) and a small, trainable IP image processing plugin layer tailored to specific IP image processing scenarios, providing the foundation for subsequent efficient parameter fine-tuning training.

[0076] A basic multimodal large model refers to a multimodal large model that has been pre-trained on large-scale, multi-type data (such as image and text pairs) and possesses general image generation capabilities. This pre-trained model acquires extensive knowledge about the world, enabling it to understand text and generate corresponding, diverse, but non-specific images. In some embodiments, the model's parameters are frozen during subsequent fine-tuning training and do not participate in updates. Examples include: mijourney large model, flux large model, mixed-source large model, Jimeng large model, Stable Diffusion v1.5 model, etc.

[0077] IP Image Processing Plugin Layer: This refers to a trainable, modular network layer designed for efficient parameter fine-tuning based on a machine learning model. Optionally, it is implemented as a Low-Rank Adaptive (LoRA) layer. This "IP Image Processing Plugin Layer" consists of multiple low-rank matrices and is injected into a specific network layer (such as the attention layer of U-Net) of the underlying multimodal large model. This plugin layer is the only part of the entire initial IP image generation model that performs parameter learning and updating; its role is specifically to learn and encode the unique visual features of the target IP image.

[0078] Schematic, a pre-trained "basic multimodal large model" weight parameters are loaded into the computing device memory. Then, the parameters of the "IP image processing plugin layer" are initialized. This plugin layer (e.g., a LoRA layer) consists of multiple low-rank matrices. These initialized plugin layers are injected into a preset network layer of the basic multimodal large model. In one embodiment, the preset network layer is a cross-attention module in the U-Net network structure of the basic model.

[0079] Finally, the parameters of the constructed composite model architecture are set for training. Specifically, all original parameters of the "basic multimodal large model" are set to a frozen state (i.e., untrainable) to protect its powerful general image generation capabilities from being compromised. Simultaneously, all parameters of the "IP image processing plugin layer" are set to a trainable state, enabling it to learn new knowledge in subsequent training steps.

[0080] After this step is completed, you will get a configured "initial IP image generation model" ready for training.

[0081] To illustrate, let's take the IP character "E Xiaotian" as an example:

[0082] Parameter settings: Basic multimodal large model: Stable Diffusion v1.5 model is selected; IP image processing plugin layer: LoRA technology is used, and its network rank is set to 8.

[0083] Execution example:

[0084] Loading steps: Load the weights of the Stable Diffusion v1.5 model in a deep learning framework (such as PyTorch).

[0085] Injection Steps: Using a library that supports efficient parameter fine-tuning, the U-Net network structure of the Stable Diffusion v1.5 model is traversed to identify all cross-attention modules. Within each cross-attention module, a LoRA plugin layer consisting of two low-rank matrices (A and B) is connected in parallel to its query and value projection matrices.

[0086] Setting up the state: The program sets the `requires_grad` attribute of all original parameters of the Stable Diffusion v1.5 model to `False` to freeze the model. Simultaneously, it sets the `requires_grad` attribute of all newly injected LoRA plugin layers to `True`, making them the only trainable parts.

[0087] Output Step: The final output of this step is a fully built, trainable composite model in memory. This model retains all the capabilities of Stable Diffusion v1.5 and adds an additional trainable LoRA plugin layer specifically for learning "Goose Xiaotian" features.

[0088] S206: Based on the target IP image, fine-tune the model parameters of the IP image processing plugin layer in the initial IP image generation model to obtain a trained reference IP image generation model.

[0089] Indicatively, the target IP image can be used to add noise to obtain a noisy IP image sample. The noisy IP image sample can then be used to train the initial IP image generation model for at least one round to obtain a denoised prediction image. Based on the denoised prediction image and the target IP image, the model processing loss during the model training process can be determined. The model processing loss can then be used to fine-tune the model parameters of the IP image processing plugin layer in the initial IP image generation model to obtain a trained reference IP image generation model.

[0090] Image noise addition: In the training process of a diffusion model, random noise (usually Gaussian noise) is added to a clear original training image according to a preset noise scheduling mechanism. This process aims to simulate the forward step of the diffusion process, providing the model with input samples for learning the denoising task.

[0091] Noisy IP Image Samples: These refer to images containing a certain level of noise obtained by "image noise enhancement" of the target IP image. These noisy samples, along with their corresponding text annotations, constitute the input to the model in a single training iteration.

[0092] Model processing loss: This refers to a numerical value used to quantify the difference between the model's prediction and the true value. In the context of diffusion model training, this loss value is typically calculated by comparing the model's denoised predictions of noisy samples with the original, uncluttered, clear image (or the added noise itself). This loss value drives the optimization of model parameters.

[0093] Model parameter fine-tuning refers to the process of calculating gradients using the backpropagation algorithm based on the calculated model processing loss, and then updating the trainable parameters in the model using an optimizer (such as AdamW). In this embodiment of the invention, this fine-tuning process is highly selective, operating only on specific "plugin layers".

[0094] In one feasible implementation, the following approach may be referenced:

[0095] First, prepare the input sample: Select a target IP image and its corresponding text annotation (such as trigger words) from the dataset prepared in step S202. Perform "image noise addition" processing on the image to generate an "IP image noise-added sample".

[0096] Start model training and perform forward propagation and prediction: Take the noisy sample and the corresponding text annotation (including IP image trigger words) as input and feed them into the initial IP image generation model constructed in step S204. Perform a forward propagation calculation and the model will output a denoised prediction result (denoised prediction image) for the noisy sample.

[0097] Loss Calculation: The denoised prediction results output by the model are compared with the original "target IP image" without noise, and the "model processing loss" for this iteration is calculated based on the preset loss function (such as mean squared error loss, MSE Loss).

[0098] Backpropagation and parameter update: Based on the model processing loss, a backpropagation algorithm is executed to calculate the gradient. Based on this gradient, "model parameter fine-tuning" is performed only on the trainable parameters of the "IP Image Processing Plugin Layer" in the "Initial IP Image Generation Model". During this process, the parameters of the "Basic Multimodal Large Model" remain frozen and are not updated.

[0099] Iteration and Completion: Repeat the above steps, traversing the entire training dataset at least once (one epoch) or reaching the preset number of training steps. After training is complete, save the parameters of the fully fine-tuned "IP Image Processing Plugin Layer". This saved parameter file, together with the basic multimodal large model, constitutes the trained reference IP image generation model.

[0100] In the embodiments described in this specification, a "reference IP image generation model" with extremely high generation fidelity for the target IP image was successfully constructed using the above method. This process trains the model using a small number of clean images and utilizes low-rank fine-tuning (LoRA) technology to accurately solidify the unique knowledge of the IP image into a trainable "plugin layer." This not only greatly reduces training costs but, more importantly, the resulting model has the ability to stably reproduce the core features of the IP image.

[0101] Optionally, the specific execution of generating at least one IP image training material based on the target IP image and the reference IP image generation model can employ one or more of the following fitting methods, as detailed below:

[0102] In one feasible implementation: multiple image scene images are generated for the target IP image, and the first training material image is obtained by generating materials based on the image scene images, the target IP image image, and the reference IP image image generation model;

[0103] This implementation illustrates a procedural decoupling strategy based on process steps, the process of which aims to separate scene generation from accurate image rendering, as follows:

[0104] In the scene generation stage: First, one or more general text-to-image generation models (such as Midjourney, Flux, etc.) are used to input text prompts describing the target scene, actions, lighting and composition, and generate an image scene diagram.

[0105] Next, in the image compositing stage: image editing tools (such as Photoshop) are used to composite the image from the previous step. Specifically, the target IP image (or its key identifying parts, such as the head) is superimposed onto the corresponding position of the main image in the image scene image through compositing, replacement, or splicing, resulting in an intermediate image with a preliminary composition but which may contain traces of compositing.

[0106] Finally, the reference IP image generation model is invoked, and the intermediate image is used as input. The reference IP image generation model will redraw the intermediate image. Utilizing its high-fidelity "overfitting" characteristic to the target IP image, the model can automatically and accurately correct and reshape the synthesized parts to perfectly match the official characteristics of the target IP image, while preserving the background and lighting atmosphere of the scene image, ultimately outputting a high-quality result.

[0107] In one feasible implementation: multiple material generation prompts are determined for the target IP image, and the reference IP image image generation model is controlled to perform hierarchical control image generation processing based on the material generation prompts to obtain a second training material image;

[0108] In illustrative terms, this implementation method is equivalent to a decoupling strategy based on the internal structure control of the model. Its processing aims to precisely separate the generation control of the IP image from the scene, as follows:

[0109] First, determine the prompt words: Based on the training requirements, determine one or more sets of "material generation prompt words" that are rich in content and diverse in scenarios.

[0110] Then, the feature layer group is located: this can be determined through experimental testing or prior knowledge, identifying which one or more consecutive layers in the U-Net network (e.g., a structure containing 17 main layers) in which the reference IP image generation model operates, is primarily responsible for generating the core features of the IP image. This layer group is called the "identity feature layer group".

[0111] Reconfiguration of layered control: In the image generation user interface or program script, the weights of the reference IP image generation model (e.g., a LoRA model using LoRA) are configured in layers. Specifically, a preset LoRA weight (e.g., weight value of 1.0) is applied only to the defined identity feature layer group, while the LoRA weights are masked (e.g., weight value of 0) on all other layers of U-Net.

[0112] Then, decoupled generation is performed: the determined prompt words are input into the layered configuration model, and image generation processing is initiated. During this process: the generation of the IP image is primarily controlled by the precisely activated LoRA weights on the "identity feature layer group," ensuring high fidelity. Scene elements such as background, composition, lighting, and style are mainly generated by other network layers in the basic diffusion model that are not affected by LoRA, guided by the text prompt words, thus ensuring scene richness and high responsiveness to the prompt words.

[0113] Finally, the training material is output: After the generation and processing are completed, the resulting image is the second training material image;

[0114] In one feasible implementation: the image generation canvas area is divided into at least one canvas image generation area, and the reference IP image generation model is controlled to perform partition control image generation processing based on the canvas image generation area to obtain a third training material image.

[0115] This implementation illustrates, illustratively, a spatial partitioning-based decoupling strategy. The processing aims to precisely control the position of individual elements in an image to generate training material with complex or specific layouts, as detailed below:

[0116] First, region division is performed: In an image generation environment that supports partition control, the complete "image generation canvas area" is divided into at least one IP character area and one or more scene areas. In one embodiment, it can be divided into two or more non-overlapping rectangular areas, and a weight is assigned to each area.

[0117] Next, partition prompts are assigned: a unique, targeted text prompt is assigned to each partitioned "canvas image generation area". Specifically, in the designated IP image area, the input prompt must include the IP image trigger word and enable the reference IP image generation model; in other scene areas, prompts that only describe the corresponding scene content are input.

[0118] Furthermore, image generation is performed by partitioning the image: During this process, the model applies different text prompts to different regions based on the partition settings. Specifically, when predicting denoising for pixels within the IP image area, the model primarily relies on the first prompt and the reference model for that region; when predicting denoising for pixels within the scene area, it primarily relies on the second prompt for that region.

[0119] Finally, the training material is output: After the generation process is complete, the model outputs the third training material image. The content of this image strictly follows the respective prompts in different regions, thereby achieving precise separation and control of the IP image and scene elements in spatial layout.

[0120] Optional, please see Figure 4 , Figure 4 This is a schematic diagram of the process for generating first training material images. Specifically, the first training material images are generated by using the image scene diagram, the target IP image image, and the reference IP image image generation model. This can be done in the following ways:

[0121] S302: Composite the head image from the target IP image onto the image scene image to form an intermediate composite image;

[0122] Head image: refers to the image portion extracted from the "target IP image" that contains the head area of ​​the target IP image. The head is usually the most core and recognizable visual feature of an IP image.

[0123] Image scene illustration: refers to an image generated using a general text-to-image generation model, which contains a main image of a non-target IP image and a specific scene, action, and posture.

[0124] Intermediate composite image: This refers to the image obtained after the "composite" process, which initially integrates the IP character's head and the scene's body posture. This image may have visual imperfections such as mismatched lighting, colors, or edges, and serves as direct input for the next step of refining the IP character image generation model.

[0125] As an illustration, the "head image" is precisely separated or extracted from the "target IP image" using image editing tools (such as Adobe Photoshop). This process can be achieved using selection tools, channel masks, and other techniques.

[0126] The extracted "head image" is then used as a new layer and overlaid on the "image scene image." Two-dimensional transformations, including but not limited to scaling, rotation, and displacement, are performed on the "head image" layer to match and align it with the body of the main image in the "image scene image" in terms of size and angle. During this process, the original head in the image scene image can be hidden or erased.

[0127] Finally, the layers processed as described above are merged, and a single image file is exported. This file is called the "intermediate composite image".

[0128] S304: The intermediate synthesized image is denoised using the reference IP image generation model to obtain the first training material image.

[0129] Image denoising: In the context of the image-to-image generation model of the reference IP image, this term refers to a guided redrawing process. Its technical essence is as follows: First, according to a set denoising intensity, a certain level of noise is added to the input image (i.e., the intermediate composite image). Then, based on this noisy image, the model, guided by optional text prompts, gradually and iteratively removes the noise, ultimately generating a completely new, corrected, and blended clear image.

[0130] The first training image is a high-quality image that seamlessly and faithfully integrates the target IP character with the new scene. This image can be directly used as training data for subsequent model optimization.

[0131] Indicatively, the S304 process aims to utilize the high-fidelity characteristics of the reference IP image generation model to automatically correct and optimize the manually synthesized image from the previous stage, as follows:

[0132] In a diffusion model environment that supports image-to-image generation, the "intermediate composite image" produced in step S302 is loaded as the input image. At the same time, the "reference IP image generation model" (i.e., the LoRA model) obtained in step S102 is loaded and activated.

[0133] Next, configure the generation parameters: set the key parameters for the image generation process, with the most crucial being the noise reduction intensity. This parameter typically ranges from 0 to 1, and it determines the degree to which the model modifies the input image. A moderate value (e.g., 0.4-0.7) allows the model sufficient space for content correction and redrawing while preserving the original image's composition and pose. Additionally, a helpful text prompt can be entered to further guide the generation direction.

[0134] By referencing the IP image generation model to perform denoising, the model will perform "image denoising" on the "intermediate composite image" according to the set denoising intensity. In this process, the model will be strongly guided by the target IP image at each step of denoising, so that the areas related to the IP image (especially the head) are reshaped and refined to perfectly match the high-fidelity features of the target IP image that it has learned.

[0135] After denoising, the model outputs a completely new image that not only retains the overall composition and novel scene of the "intermediate composite image" but also eliminates the original compositing artifacts (such as edge and lighting mismatch issues), resulting in a natural and accurate representation of the IP character. This output image serves as the "first training material image."

[0136] For example, refer to Figure 5 , Figure 5 This is a schematic diagram of a scene generated from the first training material image. Figure 5 The scenario shown demonstrates the process of generating "first training material images" through a "procedural decoupling" strategy, the core of which combines human editing with AI correction.

[0137] Figure 5 The left-hand diagram shows the "Generation of Image Scene Image," which is the preparation stage. As shown, the operator first uses a general text-based image model (e.g., Midjourney) to generate a basic "Image Scene Image." This image defines the overall composition, image pose (such as a waving gesture), and basic scene of the required material, but the image in the image is not the target IP image. This image serves as the input material for the next step, S302.

[0138] Figure 5 The intermediate image shown illustrates the "head image synthesis," a step that precisely corresponds to the S302 processing procedure. Operators use image editing tools (such as Photoshop) to precisely place the official "head image" of the target IP character (e.g., "E Xiaotian") onto the target IP character through synthesis, replacement, and other methods. Figure 1 The original figure's body in the "character scene diagram". After processing, the resulting... Figure 5 The image shown in the intermediate image is called the "intermediate composite image". At this point, the composition of the image is basically complete, but there may be visual inconsistencies in terms of light and shadow, color, and edges between the head and the body.

[0139] Figure 5 The right-hand image shows the step of "model denoising to obtain the first training image," which corresponds to the processing step S304. The operator will... Figure 2 The "intermediate composite image" is used as input, and the "reference IP image generation model" (i.e., the LoRA model that overfits the target IP image) is used to perform "image denoising processing" (i.e., image-to-image refinement). During this process, the reference model, with its high-fidelity representation of the IP image, automatically corrects and redraws the IP image's head, ensuring it blends perfectly with the body and the lighting and textures of the environment, eliminating all compositing artifacts. The final result is as follows: Figure 5 The image on the right shown is a high-quality image with consistent visual effects, accurate target IP image, and vivid posture, which is the "first training material image" that can be used for model training.

[0140] In this manual, through the manual or semi-automatic synthesis in step S302, operators can precisely control the composition, posture, and layout of the IP image in new and diverse scenes with a high degree of freedom, effectively overcoming the limitations of relying solely on text prompts for creation. Next, step S304 utilizes a high-fidelity "reference IP image generation model" for automated image refinement, seamlessly integrating synthesis traces, unifying lighting and shadow styles, and accurately restoring the core features of the IP image. This efficiently transforms the initial, potentially flawed, synthesized image into a high-quality training material with stable quality and unified visual effects.

[0141] Optional, please see Figure 6 , Figure 6 This is a schematic diagram of a process for generating second training material images. Specifically, the process of controlling the reference IP image generation model based on the material generation prompts to perform hierarchical image generation processing to obtain the second training material images can be carried out in the following ways:

[0142] S402: Input the material generation prompts into the reference IP image generation model;

[0143] S404: Determine multiple first network layers to be activated and at least one second network layer to be masked through the reference IP image generation model, and generate second training material images by using the first network layers.

[0144] Material generation prompts: These are text inputs specifically written for generating training materials, containing rich scene descriptions. In addition to IP character trigger words, these prompts also define diverse elements such as scenes, actions, and styles in detail.

[0145] The first network layer refers to the group of network layers in the U-Net network structure of the reference IP image generation model that are primarily responsible for generating the core identity features of the target IP image. In the hierarchical control processing, these layers are to be activated; that is, the target IP image generation control weights of the reference IP image generation model will be applied to these layers.

[0146] The second network layer refers to the remaining network layers in the U-Net network structure besides the first network layer. In the layered control processing, these layers are to be masked, meaning that the target IP image control generation weights (which can be called LoRA weights) will not apply to these layers, and their generated content will be mainly guided by the material generation prompts and the basic large model.

[0147] Layered control image generation processing: This refers to a technical process that decouples the IP image and the scene in the internal structure of the model by activating the first network layer to control the generation of the target IP image and shielding the second network layer from controlling the generation of the target IP image.

[0148] In a schematic manner, a pre-written "material generation prompt word" containing IP character trigger words and rich scene descriptions is input into a configured reference IP character image generation model. Before execution, the reference IP character image generation model (LoRA model) is configured with hierarchical weights. Specifically, LoRA weights are applied only to a defined "first network layer," while the LoRA weights on the "second network layer" are set to invalid (e.g., weight value of 0). Image generation is then initiated.

[0149] During the iterative denoising process of the model:

[0150] Under the strong guidance of LoRA weights, the "first network layer" is responsible for accurately generating the identity features of the target IP image.

[0151] The "second network layer," without being influenced by LoRA weights, guides the basic large model to generate rich scene, lighting, and composition details entirely based on "material generation prompts."

[0152] Finally, the reference IP image generation model outputs a "second training material image" that accurately represents the target IP image and is rich in scene details.

[0153] For example, such as Figure 6a ,like Figure 6b And such as Figure 6c , Figure 6a ,like Figure 6b And such as Figure 6c This is a set of layered control image generation processing effect images. Taking the target IP image "E Xiaotian" as an example, and based on a reference IP image image generation model trained on the SD1.5 model, the target IP image control generation weights (which can be called LoRA weights) are applied to the Unet network of the reference IP image image generation model. There are 17 layers in total, and these 17 layers can be loaded with different LoRA weights to achieve different effects, rather than simply assigning a uniform weight. Through creative work, the corresponding image output details for each LoRA layer have been determined. Figure 6a ,like Figure 6b And such as Figure 6c It's the LoRA layered control effect from E Xiaotian;

[0154] You can see Figure 6aThe first set of images on the left illustrates the state of "full application of LoRA" for controlling the generation weights of the target IP image (which can be called LoRA weights). The left image shows that the weights (all weight values ​​are 1) from the reference IP image generation model are applied to all 17 layers of the U-Net network. This is equivalent to no layered control. In the generated image, the image of "Little Goose" has extremely high fidelity, but the background is extremely monotonous and homogeneous, completely failing to reflect the rich scene that the prompt should have. This perfectly demonstrates the "overfitting" state of the "reference IP image generation model" when it is uncontrolled.

[0155] You can see Figure 6a The right-hand diagram in the first group illustrates the target IP image generation weights (in the state "LORA not applied"). The left-hand diagram shows that when the weights in the reference IP image generation model are set to 0 (all weight values ​​are 0), the IP image processing plugin layer responsible for controlling the generation of the target IP image is completely disabled. Only the basic multimodal large model is used for generation. The generated image is a detailed ordinary goose / duck that matches the scene description in the prompt, but it completely lacks the IP characteristics of "Goose Xiaotian". This demonstrates the powerful scene generation capabilities of the basic large model, while also proving its limitations for specific IP images.

[0156] You can see Figure 6b The left-hand diagram illustrates that "the target IP image generation has low correlation with the first few layers." It activates the weights of the first five layers of the U-Net network (weights are 1, 1, 1, 1, 1, 0, ...), while blocking all the middle and later layers. As a result, the generated image does not show the characteristics of the target IP "Goose Xiaotian" at all, but generates a normal, realistic goose. The quality of the background scene generation is acceptable.

[0157] You can see Figure 6b The right-hand side of the diagram illustrates the correlation between the target IP image generation and the three middle layers, activating the weights of the model in the middle part of the U-Net network. The result: the generated image quality is excellent. Not only is the "Goose Little Sky" IP image accurately and faithfully reproduced, but its posture, clothing, and surrounding scene are also very vivid, diverse, and well-integrated.

[0158] Figure 6b The comparison of the two figures shows that the core identity features of the target IP image are mainly controlled by the intermediate layers of the U-Net network. By precisely activating these "identity feature layers," the rest of the model can freely generate rich scenes while ensuring that the image is not distorted.

[0159] Figure 6C shows the activation model in the middle 6 layers of the U-Net network, and then the other layers are masked. This ensures that the target IP image "E Xiaotian" is correct, while the background better matches the description of the prompt.

[0160] This specification describes a refined strategy that controls the internal structure of the model to achieve precise decoupling between IP character generation and scene generation. This method identifies and selectively activates the "first network layer," responsible only for the IP character's identity features, while simultaneously shielding the remaining "second network layers." This allows the high fidelity of the reference model and the high generalization ability of the base model to function in parallel within the same generation task. This fundamentally solves the technical challenge of the reference model's inability to generate complex backgrounds due to overfitting. It enables the efficient and stable generation of a large amount of high-quality training material with accurate IP characters and scene details perfectly faithful to the text prompts, providing crucial data support for training a superior target model.

[0161] Optional, please see Figure 7 , Figure 7 This is a schematic diagram of a process for generating a third training material image. Specifically, the process involves dividing the image generation canvas area into at least one canvas image generation region, and controlling the reference IP image generation model to perform partitioning control image generation processing based on the canvas image generation region to obtain the third training material image. This can be achieved in the following ways:

[0162] S502: Divide the image generation canvas area into at least one IP image area and one scene area;

[0163] IP Image Area: This refers to a specific sub-area within the image generation canvas area, defined by coordinates, masks, or other methods, specifically used to generate the target IP image.

[0164] Scene area: refers to one or more sub-areas within the image generation canvas area, excluding the IP image area, specifically used to generate backgrounds, environments, or other scene elements independent of the IP image.

[0165] S504: Control the reference IP image generation model to perform partitioned image generation processing based on the IP image area and the scene area to obtain the third training material image.

[0166] Partition-based image generation processing refers to a guided diffusion model that generates content in different areas of the canvas according to different instructions. This process achieves precise control over the final image spatial layout and composition by assigning independent text prompts to IP character areas and scene areas.

[0167] In a schematic example, within an image generation environment that supports partitioned control, the complete image generation canvas area is logically or in the form of a mask divided into at least two regions, defined as the "IP Image Region" and the "Scene Region," respectively. Then, an independent, targeted text prompt is assigned to each partitioned region. Specifically: in the designated "IP Image Region," the input prompt must contain an IP image trigger word and enable the aforementioned reference IP image generation model (LoRA model); in the "Scene Region," the input prompt only describes the corresponding scene content.

[0168] Then, image generation processing is initiated. During this process, the reference IP image generation model applies different text prompts to different regions based on the partition settings. Specifically, when the model performs denoising prediction for pixels within the "IP image region," it primarily calculates based on the first prompt corresponding to that region and the reference model; while when performing denoising prediction for pixels within the "scene region," it primarily calculates based on the second prompt corresponding to that region.

[0169] After the generation process is complete, the model outputs a single, complete image. The content of this image strictly follows the respective prompts in different areas, thus achieving precise separation and control of the IP image and scene elements in spatial layout. This image serves as a high-quality "third training material image."

[0170] For example, such as Figure 8 As shown, Figure 8 This is a schematic diagram of a scene divided into regions. Figure 8 It demonstrates how a complete "image generation canvas area" can be divided into multiple functionally independent "canvas image generation areas", and each area is given a preset independent generation instruction (i.e., text prompt).

[0171] The S502 reflects the regional division, such as Figure 8 As shown, the entire canvas is divided into five distinct regions (numbered 0 to 4). This segmentation process is a concrete example of "dividing the image generation canvas area into at least one IP character area and one scene area" as described in S502. In this example:

[0172] The IP character area is the blue central area labeled "3 Little Goose sitting on the ground".

[0173] Scene area: This refers to the area surrounding the icons marked "0 trees", "1 day", "2 wildflowers", and "4 wildflowers".

[0174] Furthermore, the instructions for allocating partition control (S504 preparation): Figure 8The text in the code represents the individual text prompts to be assigned to each region. This is a prerequisite for executing S504. On one hand, for the "IP Image Region" (Region 3), the prompt "Little Goose is sitting on the ground" containing the trigger word is assigned, which will guide the reference model to generate the IP image in this region. On the other hand, for each "Scene Region," prompts describing the corresponding scene elements are assigned, such as "tree," "sky," and "field flowers." The generation of these regions will not be affected by the IP image LoRA model.

[0175] Figure 8 The "preset layout" diagram shown is a blueprint for the partitioning control strategy, from abstract concept to concrete implementation. It illustrates the region partitioning action of S502 and provides a clear and visual instruction set for partition generation in S504. When subsequent models begin generation, they are guided to refer to this layout to generate trees in the orange area, sky in the red area, and "little geese" in the blue central area, thus obtaining a high-quality "third training material image" with a composition that perfectly conforms to the preset layout.

[0176] In one feasible implementation, the following approach can be referenced:

[0177] Step B2: Divide the image generation canvas area into at least one IP image area and one scene area, and determine a first image prompt word for the IP image area and a second scene prompt word for the scene area;

[0178] Step B4: Control the reference IP image generation model to generate a first image region image in the IP image region based on a first image prompt, and generate a second scene region image in the scene region based on a second scene prompt. Perform image fusion processing on the first image region image and the second scene region image to obtain a third training material image.

[0179] First image area image / Second scene area image: These two terms describe the intermediate products of the partitioning process, namely, the content conceptually generated in their respective areas based on their respective prompts before final combination.

[0180] Image fusion processing: refers to the process of seamlessly integrating the contents of the "first image region image" and the "second scene region image" onto the same canvas during the denoising process of the diffusion model, ultimately forming a single, complete image with harmonious composition.

[0181] In a schematic example, within an image generation environment that supports partitioned control, the entire image generation canvas area is divided into an "IP Image Area" and a "Scene Area." Next, separate text prompts are determined and entered for each of these two areas. Specifically, a "First Image Prompt" (including a trigger word) is entered for the "IP Image Area," and a "Second Scene Prompt" is entered for the "Scene Area."

[0182] Initiate image generation processing. During this process, the image generation model for the reference IP image is guided by a partitioning control mechanism:

[0183] A parallel generation method is adopted: within the "IP image area", the reference IP image generation model is controlled to generate the content of the "first image area image" mainly based on the "first image prompt word"; at the same time, within the "scene area", the content of the "second scene area image" is generated mainly based on the "second scene prompt word".

[0184] Then, image fusion is performed: In each step of the iterative denoising process of the reference IP image generation model, the model considers region boundaries and global information to ensure that the content generated in different regions can be seamlessly and naturally fused into a single image. After completion, the "third training material image" is obtained.

[0185] For example, such as Figure 9 As shown, Figure 9 This is a comparison chart of the generation effects of a partition control method. Figure 9 By showcasing the generation results using two methods—"using a single prompt word" and "using zone control"—the image generation effect is clearly revealed. Figure 9 The image on the left shows the "simple prompt word generation effect," which is generated directly using traditional, single text prompt words without the aforementioned partitioning control steps. The generation results are highly unstable and inconsistent in quality. This is reflected in the uncontrolled composition: the position and size of the IP character "Goose Little Sky" in the image are unpredictable, sometimes excessively large and filling the entire frame, and sometimes misaligned; and the image is flawed: in the lower right corner image, a serious generation error occurs, with only "Goose Little Sky's" head generated on the ground, the body missing—the so-called "image collapse."

[0186] Figure 9The image on the right shows the effect achieved using partition control, following the steps of the partition control method described above. First, the canvas is divided into an "IP image area" and a "scene area" (corresponding to B2), then independent prompts are assigned to each area for partition control generation (corresponding to B4). Analysis of the results shows a significant leap in the quality and stability of the generated results compared to the left. This demonstrates precise image reconstruction: in all four images, "Little Goose" is accurately placed in a reasonable position within the scene (e.g., sitting in a flowerbed), and its size and proportion to the background are very harmonious. Furthermore, the image is stable: there are no instances of image distortion; each image is high-quality and usable.

[0187] The above method fundamentally solves the problem of insufficient spatial understanding. The model is precisely instructed on which area to generate the IP image and which area to generate the background, thus enabling it to stably produce "third training material images" with beautiful composition and correct content.

[0188] In this specification, the partitioning control strategy comprised of steps S502 to S504 fundamentally improves the technical pain point of weak spatial layout understanding commonly found in diffusion models. By pre-dividing the canvas into regions in S502 and applying independent generation instructions to different regions in S504, this method provides the model with a clear and explicit composition generation method, effectively avoiding serious problems such as uncontrollable IP image positioning, improper subject-to-background ratio, and even image content "collapse" that are common when using a single prompt word. Ultimately, this strategy can ensure the stable and efficient generation of a large number of training materials with accurate composition, reasonable layout, and high quality, greatly improving the success rate and output quality of the data augmentation process.

[0189] The following will combine Figure 10 This specification provides a detailed description of the image processing apparatus provided in the embodiments. It should be noted that... Figure 10 The image processing device shown is used to perform the functions described in this manual. Figures 1-9 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-9 The example shown.

[0190] Please see Figure 10 This diagram illustrates the structure of an image processing device according to an embodiment of this specification. The image processing device 1 can be implemented as all or part of a user terminal through software, hardware, or a combination of both. According to some embodiments, the image processing device 1 includes a material acquisition module 11 and a model fine-tuning module 12, specifically used for:

[0191] Material acquisition module 11 is used to acquire customized target IP image images, and to use the target IP image images to perform low-rank fine-tuning training of the first diffusion model to obtain a reference IP image generation model.

[0192] The material acquisition module 11 is used to generate training materials for at least one IP image generation training material for the reference IP image generation model by employing at least one image and scene decoupling generation strategy.

[0193] The model fine-tuning module 12 is used to perform low-rank fine-tuning training of the reference IP image generation model based on the IP image generation training material and the target IP image image to obtain the target IP image generation model.

[0194] In one feasible implementation, the step of obtaining a customized target IP image and using the target IP image to perform low-rank fine-tuning training of a first diffusion model to obtain a reference IP image generation model includes:

[0195] Obtain a preset number of target IP image images of a preset background film type, and annotate the target IP image images with IP image trigger words;

[0196] An initial IP image generation model is constructed, which includes a basic multimodal large model and an IP image processing plugin layer for IP image generation scenarios.

[0197] Based on the target IP image, the model parameters of the IP image processing plugin layer in the initial IP image generation model are fine-tuned to obtain a trained reference IP image generation model.

[0198] In one feasible implementation, the step of fine-tuning the model parameters of the IP image processing plugin layer in the initial IP image generation model based on the target IP image to obtain a trained reference IP image generation model includes:

[0199] The target IP image image is used to add noise to obtain a noise-added IP image sample;

[0200] The initial IP image generation model is trained at least once using the noisy sample of the IP image to obtain a denoised prediction image. Based on the denoised prediction image and the target IP image, the model processing loss during the model training process is determined. The model processing loss is used to fine-tune the model parameters of the IP image processing plugin layer in the initial IP image generation model to obtain a trained reference IP image generation model.

[0201] In one feasible implementation, the step of generating at least one IP image generation training material based on the target IP image and the reference IP image generation model includes:

[0202] Multiple image scene images are generated for the target IP image. Based on the image scene images, the target IP image images, and the reference IP image image generation model, the first training material image is generated.

[0203] In one feasible implementation, the step of generating at least one IP image generation training material based on the target IP image and the reference IP image generation model includes:

[0204] Multiple material generation prompts are determined for the target IP image, and the reference IP image image generation model is controlled to perform hierarchical control image generation processing based on the material generation prompts to obtain the second training material image;

[0205] In one feasible implementation, the step of generating at least one IP image generation training material based on the target IP image and the reference IP image generation model includes:

[0206] The image generation canvas area is divided into at least one canvas image generation area, and the reference IP image generation model is controlled to perform partitioning control image generation processing based on the canvas image generation area to obtain the third training material image.

[0207] In one feasible implementation, the step of generating the first training material image based on the image scene diagram, the target IP image image, and the reference IP image image generation model includes:

[0208] The head image from the target IP image is composited onto the image scene image to form an intermediate composite image;

[0209] The intermediate synthesized image is denoised using the reference IP image generation model to obtain the first training material image.

[0210] In one feasible implementation, the step of controlling the reference IP image generation model to perform hierarchical control image generation processing based on the material-generated prompt words to obtain the second training material image includes:

[0211] The material generation prompts are input into the reference IP image generation model. The reference IP image generation model determines multiple first network layers to be activated and at least one second network layer to be blocked. The IP image is generated through the first network layer to obtain the second training material image.

[0212] In one feasible implementation, dividing the image generation canvas area into at least one canvas image generation region, and controlling the reference IP image generation model to perform partitioned image generation processing based on the canvas image generation region to obtain a third training material image, includes:

[0213] The image generation canvas area is divided into at least one IP image area and one scene area. The reference IP image image generation model is controlled to perform partitioning control image generation processing based on the IP image area and the scene area to obtain the third training material image.

[0214] In one feasible implementation, dividing the image generation canvas area into at least one IP character area and one scene area, and controlling the reference IP character image generation model to perform partitioned image generation processing based on the IP character area and the scene area to obtain a third training material image, includes:

[0215] The image generation canvas area is divided into at least one IP image area and one scene area, and a first image prompt word for the IP image area and a second scene prompt word for the scene area are determined.

[0216] The reference IP image generation model is controlled to generate a first image region image based on a first image prompt in the IP image region, and a second scene region image based on a second scene prompt in the scene region. The first image region image and the second scene region image are then fused to obtain a third training material image.

[0217] It should be noted that the image processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the image processing method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device and the image processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process can be found in the method embodiments, which will not be repeated here.

[0218] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0219] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-9 The image processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-9 The specific details of the illustrated embodiments will not be elaborated here.

[0220] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-9 The image processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-9 The specific details of the illustrated embodiments will not be elaborated here.

[0221] Please refer to Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 1010, a memory 1020, an input device 1030, an output device 1040, and a bus 1050. The processor 1010, memory 1020, input device 1030, and output device 1040 may be connected to each other via the bus 1050.

[0222] Processor 1010 may include one or more processing cores. Processor 1010 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 1020, and by calling data stored in memory 1020. Optionally, processor 1010 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 1010 may integrate one or more of a central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 1010 and may be implemented separately through a communication chip.

[0223] The memory 1020 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1020 may include non-transitory computer-readable storage medium. The memory 1020 may be used to store instructions, programs, code, code sets, or instruction sets.

[0224] The input device 1030 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 1040 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In this embodiment, the input device 1030 can be a temperature sensor for acquiring the operating temperature of the electronic device. The output device 1040 can be a speaker for outputting audio signals.

[0225] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WIFI) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0226] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0227] exist Figure 11 In the electronic device, the processor 1010 can be used to call a program stored in the memory 1020 and execute it to implement the image processing method as described in the various method embodiments of this specification.

[0228] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0229] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. An image processing method, characterized in that, The method includes: Obtain a customized target IP image, and use the target IP image to perform low-rank fine-tuning training of the first diffusion model to obtain a reference IP image generation model; At least one IP image generation training material is obtained by generating training materials based on the target IP image and the reference IP image generation model. Based on the IP image generation training materials and the target IP image, the reference IP image generation model is trained by performing low-rank fine-tuning of the second diffusion model to obtain the target IP image generation model.

2. The method according to claim 1, characterized in that, The process of generating at least one IP image generation training material based on the target IP image and the reference IP image generation model includes: Multiple image scene images are generated for the target IP image. Based on the image scene images, the target IP image images, and the reference IP image image generation model, the first training material image is generated.

3. The method according to claim 2, characterized in that, The process of generating the first training material image based on the image scene diagram, the target IP image image, and the reference IP image image generation model includes: The head image from the target IP image is composited onto the image scene image to form an intermediate composite image; The intermediate synthesized image is denoised using the reference IP image generation model to obtain the first training material image.

4. The method according to claim 1, characterized in that, The process of generating at least one IP image generation training material based on the target IP image and the reference IP image generation model includes: Multiple material generation prompts are determined for the target IP image. Based on the material generation prompts, the reference IP image image generation model is controlled to perform hierarchical control image generation processing to obtain the second training material image.

5. The method according to claim 4, characterized in that, The process of generating prompts based on the source material to control the reference IP image generation model to perform hierarchical image generation processing to obtain the second training source image includes: The material generation prompts are input into the reference IP image generation model. The reference IP image generation model determines multiple first network layers to be activated and at least one second network layer to be blocked. The IP image is generated through the first network layer to obtain the second training material image.

6. The method according to claim 1, characterized in that, The process of generating at least one IP image generation training material based on the target IP image and the reference IP image generation model includes: The image generation canvas area is divided into at least one canvas image generation area, and the reference IP image generation model is controlled to perform partitioning control image generation processing based on the canvas image generation area to obtain the third training material image.

7. The method according to claim 6, characterized in that, The step of dividing the image generation canvas area into at least one canvas image generation region, and controlling the reference IP image generation model to perform partitioned image generation processing based on the canvas image generation region to obtain the third training material image, includes: The image generation canvas area is divided into at least one IP image area and one scene area. The reference IP image image generation model is controlled to perform partitioning control image generation processing based on the IP image area and the scene area to obtain the third training material image.

8. The method according to claim 7, characterized in that, The step of dividing the image generation canvas area into at least one IP image area and one scene area, and controlling the reference IP image generation model to perform partitioning control image generation processing based on the IP image area and the scene area to obtain the third training material image includes: The image generation canvas area is divided into at least one IP image area and one scene area, and a first image prompt word for the IP image area and a second scene prompt word for the scene area are determined. The reference IP image generation model is controlled to generate a first image region image based on a first image prompt in the IP image region, and a second scene region image based on a second scene prompt in the scene region. The first image region image and the second scene region image are then fused to obtain a third training material image.

9. The method according to claim 1, characterized in that, The process of obtaining a customized target IP image and using the target IP image to perform low-rank fine-tuning training of the first diffusion model to obtain a reference IP image generation model includes: Obtain a preset number of target IP image images of a preset background film type, and annotate the target IP image images with IP image trigger words; An initial IP image generation model is constructed, which includes a basic multimodal large model and an IP image processing plugin layer for IP image generation scenarios. Based on the target IP image, the model parameters of the IP image processing plugin layer in the initial IP image generation model are fine-tuned to obtain a trained reference IP image generation model.

10. The method according to claim 9, characterized in that, The step of fine-tuning the model parameters of the IP image processing plugin layer in the initial IP image generation model based on the target IP image to obtain a trained reference IP image generation model includes: The target IP image image is used to add noise to obtain a noise-added IP image sample; The initial IP image generation model is trained at least once using the noisy sample of the IP image to obtain a denoised prediction image. Based on the denoised prediction image and the target IP image, the model processing loss during the model training process is determined. The model processing loss is used to fine-tune the model parameters of the IP image processing plugin layer in the initial IP image generation model to obtain a trained reference IP image generation model.

11. An image processing device, characterized in that, The device includes: The material acquisition module is used to acquire customized target IP image images, and to use the target IP image images to perform low-rank fine-tuning training of the first diffusion model to obtain a reference IP image generation model. The material acquisition module is used to generate training materials for at least one IP image generation training material for the reference IP image generation model by employing at least one image and scene decoupling generation strategy. The model fine-tuning module is used to perform low-rank fine-tuning training of the reference IP image generation model based on the IP image generation training materials and the target IP image to obtain the target IP image generation model.

12. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are adapted to be loaded by a processor and executed as method steps as claimed in any one of claims 1 to 10.

13. A computer program product storing at least one instruction, said at least one instruction being loaded by a processor and executing the method steps of any one of claims 1 to 10.

14. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 10.

Images