AI model automatic generation method

Abstract

The invention discloses a method for automatically generating an AI model, which belongs to the technical field of image generation and comprises the following steps: acquiring basic characteristics, posture information and clothing description of the AI model; a model image is generated through a diffusion converter according to the basic features, the clothing description and the posture information, the model image is optimized through an optimizer, and the optimized model image is obtained; wherein the diffusion converter is optimized by combining LoRA; carrying out background fusion and adjustment on the optimized model image to obtain a final model image of the AI model; according to the technical scheme, the high-quality customized AI model image can be efficiently generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and image generation, and in particular relates to an automatic generation method of an AI model. Background Art

[0002] In current e-commerce design, the effect of product display has an important impact on consumers' purchasing decisions. The traditional e-commerce model shooting process requires the collaboration of professional models, photographers, makeup artists, etc., which is not only costly but also time-consuming. With the increasing diversification and personalized needs of the e-commerce market, traditional model shooting is difficult to respond quickly to market changes and meet the design needs of different styles and scenes.

[0003] In recent years, AI technology has made significant progress in the field of image generation, and a variety of AI model generation tools have emerged. Although these tools have improved design efficiency to a certain extent, they still have the following shortcomings: • Limited generation quality: AI-generated model images cannot fully meet the high standards of e-commerce display in terms of details, realism and natural integration.

[0004] •Low degree of customization: Users lack flexibility and control when customizing model characteristics (such as gender, age, posture, clothing, etc.), making it difficult to achieve highly personalized model generation.

[0005] •Poor integration: Existing AI model generation tools often exist as independent applications and lack seamless integration with software commonly used by designers (such as Photoshop). Frequent tool switching is required, affecting design efficiency.

[0006] •Low computational efficiency: Existing methods usually require more reasoning steps, resulting in slow generation speed and difficulty in meeting real-time design requirements.

[0007] Therefore, there is an urgent need for a method that can efficiently generate high-quality, customized AI models and can be seamlessly integrated into designers' commonly used software to improve the efficiency and display effects of e-commerce design and meet diverse market needs. Summary of the invention

[0008] In order to solve the above technical problems, the present invention proposes an AI model automatic generation method to solve the problems existing in the above prior art. The present invention is applicable to embedded design software plug-ins. The method can efficiently generate high-quality, customized virtual model images, and significantly improve the generation speed and image detail fidelity through multi-model collaboration and efficient parameter training.

[0009] To achieve the above object, the present invention provides an AI model automatic generation method, comprising: Obtain the basic features, posture information and clothing description of the AI ​​model; Generate a model image according to basic features, clothing description and posture information through a diffusion transformer, and optimize the model image through an optimizer to obtain an optimized model image; wherein the diffusion transformer is optimized by combining LoRA; The optimized model image is subjected to background fusion and adjustment to obtain the final model image of the AI ​​model.

[0010] Optionally, basic features, posture information and clothing description of the AI ​​model are obtained through an interactive interface, wherein the interactive interface includes input prompts and input methods for the basic features, posture information and clothing description of the AI ​​model, and the interactive interface is also used to interactively display the model image, clothing image, fused image and final image; The basic characteristics include gender, age, skin color and expression; the clothing description includes clothing style and accessories information.

[0011] Optionally, the process of generating the model image includes: extracting input data from a model database according to the basic features, wherein the model database stores input data associated with the basic features; The input data and the clothing description are processed by the SDXL model to obtain an initial model image; wherein the U-net backbone network structure is replaced by a diffusion transformer in the SDXL model, and the clothing description is input by a cross-attention mechanism, wherein the output of the cross-attention mechanism is combined with the input data as the input of the diffusion transformer; Performing pose estimation on the initial model image through the OpenPose model to obtain body key points; Adjusting the initial model image according to body key points and posture information using a ControlNet model, wherein the ControlNet model uses a residual connection attention mechanism to connect with a cross attention layer in a diffusion transformer in the SDXL model; The adjusted model image is segmented through the SAMSegment model to obtain the model image.

[0012] Optionally, the process of generating the clothing image includes: The clothing description is encoded and transformed through CLIP text encoding to obtain a conditional vector, and the conditional vector and the noise vector of the SDXL model are used as inputs of a cross-attention mechanism. The output of the cross-attention mechanism is combined with input data as inputs of a diffusion transformer.

[0013] Optionally, the process of generating the optimized model image includes: The model image is adjusted in detail using the Inpaint model to obtain an optimized model image.

[0014] Optionally, the process of optimizing the diffusion converter by combining LoRA includes: G-LoRA and MP-LoRA are introduced into the MM-Attention block in the Transformer block in the diffusion transformer, respectively, and the states of G-LoRA and MP-LoRA are controlled by LoRA-switches, wherein the states include disabled, frozen, and trained; The G-LoRA and MP-LoRA are respectively added to the QKV linear layers in the MM-Attention block of the diffusion transformer, the G-LoRA is used to adjust the weights of the Q linear layer and the K linear layer, and the MP-LoRA is used to adjust the weight of the V linear layer.

[0015] Optionally, the process of performing background fusion and adjustment on the fused image includes: A background image is obtained, and a fused image is fused according to the background image to obtain a background fused image. The background fused image is adjusted to obtain a final model image of the AI ​​model, wherein the adjustment includes adjustment of lighting, color tone, and perspective relationship.

[0016] Optionally, the final model image generation also includes: If the final model image does not meet the requirements, the basic features, posture information and clothing description of the AI ​​model are readjusted, and the final model image is generated based on the readjusted data.

[0017] Compared with the prior art, the present invention has the following advantages and technical effects: High-quality model image generation: Using the self-trained Stable Diffusion XL (SDXL) model, we can generate detailed and realistic model images to ensure that the model features are highly consistent with user needs.

[0018] Precise posture and motion control: Apply posture estimation technology and key point detection algorithms to accurately adjust the model's body posture, ensure natural movements, and enhance the realism and dynamic expression of the image.

[0019] Natural integration of clothing and accessories: Through SDXL's Inpaint model, the clothing is seamlessly integrated with the model's body, and the clothing's wrinkles, textures, and light and shadow effects are carefully processed to ensure realism.

[0020] Natural fusion and optimization of background: Use the Inpaint model to naturally blend the model image with the background image, adjust the lighting, tone and perspective relationship, and generate a highly realistic and attractive final image.

[0021] Systematized and automated workflow: Integrate multiple models and processing nodes to automate the entire process from model feature input to image generation, reduce manual intervention, and improve generation speed and consistency.

[0022] Parameter efficiency and computation efficiency: Through LoRA technology and distillation diffusion method, the computational cost of training and inference is significantly reduced, and the overall efficiency of the system is improved.

[0023] Diversity and flexibility: Support multi-dimensional and refined model customization to meet the diverse needs of different users and enhance the personalization and uniqueness of the design.

[0024] In summary, the AI ​​model automatic generation method based on diffusion transformer of the present invention can efficiently generate high-quality, customized virtual model images through multi-model collaboration, efficient parameter training and reasoning step optimization, significantly improve the production efficiency and visual expression of e-commerce design, and meet the market demand for diversified and high-fidelity model images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] The present invention aims to provide an efficient and customized AI model automatic generation method based on diffuse transformer (DiT). Through multi-model collaborative work, efficient parameter training and reasoning step optimization, it solves the problems of insufficient generation quality, low customization, poor integration and low computational efficiency in the prior art, and improves the efficiency and display effect of e-commerce design.

[0029] The present invention provides an AI model automatic generation method based on a diffusion transformer, comprising the following steps: S1. Model feature input: User interface interaction: Design a user interface for inputting and selecting basic features of models. Through the plug-in interface embedded in the design software (such as Photoshop), users input or select basic features of models, including gender, age, skin color, etc. Support multiple input methods, such as drop-down menus, sliders or text input, to meet the operating habits of different users.

[0030] Multi-dimensional feature customization: The user further selects or inputs detailed features such as the model's posture, clothing style, accessories, etc.

[0031] The clothing description input by the user is converted into a conditional vector through text encoding, which is used for subsequent image generation and control to achieve customized clothing generation.

[0032] The system provides preset options and custom input functions to achieve multi-dimensional and refined model customization.

[0033] For the input posture data, the posture information is processed using preprocessors such as OpenPose, and the GroundingDINO and SAM Segment models are used to perform posture estimation, key point detection, and image area segmentation to generate corresponding masks and posture adjustment parameters.

[0034] S2. Model data processing Data extraction: Based on the basic feature parameters input by the user, the system extracts the corresponding model data from the pre-built multi-dimensional model database to ensure the diversity and richness of the generated models.

[0035] Image Generation: Use the self-trained Stable Diffusion XL (SDXL) model to generate model images that match the characteristics, ensuring that the model's appearance and posture are natural and realistic.

[0036] S3. Posture and movement adjustment Pose Estimation Technology: Apply pose estimation technology to adjust the model's posture based on the pose adjustment parameters selected by the user, ensuring that the movements are natural and meet the needs of specific scenes.

[0037] Combined with the model information selected by the user, precise posture adjustment is achieved through multiple depth, line drawings, and OpenPose's ControlNet model.

[0038] Key point detection and adjustment algorithm: The key point detection algorithm is used to identify the key joints in the model image, and the posture adjustment algorithm is used to generate the expected dynamic model posture to ensure that the movement conforms to the laws of physics.

[0039] S4. Clothing and accessories generation Automatic generation or adjustment of clothing: Automatically generate or adjust the model's clothing and accessories based on the clothing description, clothing style and accessories selected by the user, ensuring the natural integration of clothing and model's posture.

[0040] Image synthesis technology: Apply SDXL's Inpaint model to achieve seamless integration of clothing and model's body, and process clothing's wrinkles, texture, light and shadow effects to ensure realism.

[0041] S5. Background fusion and optimization Background image generation or selection: Automatically generate or select suitable background images based on the specific scene requirements of e-commerce display.

[0042] Image fusion and optimization: Naturally blend the generated model image with the background image, adjust the lighting, tone and perspective relationship to ensure visual consistency and coordination.

[0043] S6. Multi-model collaboration and efficient parameter training Multi-model collaboration: Combine VAE, Stable Diffusion XL, ControlNet, GroundingDINO, SAM and other models to achieve high-quality image generation and editing.

[0044] Parameter efficient training: The LoRA (Low-Rank Adaptation) technology is used to perform parameter-efficient joint training, which only adds a small number of trainable parameters and improves training efficiency.

[0045] Scientific principles of technical solutions The present invention is based on the image generation technology of the diffusion transformer (DiT), which achieves high-quality model image generation through multi-model collaboration and efficient parameter training. Stable Diffusion XL (SDXL) ensures rich details and high fidelity of model images through the step-by-step generation process of the diffusion model. Pose estimation technology uses deep learning models (such as OpenPose) to detect and adjust the key points of the model's body to ensure natural posture. LoRA technology is used for efficient joint training of parameters, reducing the computational cost of model training and inference. The distillation diffusion method improves the generation speed by optimizing the inference steps to meet real-time design needs.

[0046] Key technologies and innovations used in this invention: No additional reference network or image encoder is required: Leveraging the inherent backbone of DiT, we achieve seamless integration of conditional image inputs, simplifying the model structure and reducing computational complexity.

[0047] Multi-task collaborative training: G-LoRA and MP-LoRA are used to handle subject generation and image restoration tasks respectively. Through the LoRA-switchoRA technology, only a small number of trainable parameters are added, which realizes efficient joint training and significantly reduces training time and memory consumption.

[0048] High detail retention and natural fusion: Through SDXL's Inpaint model, the clothing is seamlessly combined with the model's body, and the details and light and shadow effects are processed to ensure the realism and coordination of the image.

[0049] The above technical solution is described in detail: Automatic generation of multi-dimensional AI models based on diffusion transformer Step 1: Model feature input The designer opens this system plug-in in Photoshop and selects the "AI model generation based on diffusion transformer" function.

[0050] In the pop-up interface, the designer selects basic characteristics such as the model's gender (such as female), age (25 years old), skin color (Asian), etc.

[0051] Further select clothing descriptions such as the model’s posture (standing), clothing (casual wear), and accessories (e.g., hat, glasses).

[0052] The clothing description input by the user is converted into a conditional vector through CLIP text encoding for subsequent image generation and control.

[0053] The above-mentioned embedded plug-in development: Develop embedded plug-ins specifically for design software (such as Photoshop) to achieve deep integration of AI model generation tools and design environments, so that designers can call and use AI model generation functions without leaving the work interface. Provide real-time preview and interactive adjustment in the interactive interface of the design software: Provide real-time preview function in the design software, allowing designers to make interactive adjustments during the generation process to ensure that the final generated image meets the design requirements.

[0054] Setting plug-in interface and interaction mechanism in the interactive interface: By designing the plug-in interface of the software, two-way data transmission and interaction with the AI ​​model generation system can be realized to ensure the smoothness and efficiency of the generation process.

[0055] Real-time preview and adjustment function: The integrated real-time preview module allows designers to instantly view and adjust model images during the generation process, improving user experience and design efficiency.

[0056] For the above basic features, posture information and clothing description, as the multi-dimensional feature input of the subsequent data processing flow, it provides an input method covering multiple parameters such as gender, age, skin color, expression, posture, clothing and accessories, so that users can achieve highly personalized model generation through an intuitive interface. At the same time, text encoding and conditional vector generation are set for clothing description: the clothing description entered by the user is converted into a conditional vector through CLIP text encoding, which accurately guides the image generation process and realizes customized clothing generation.

[0057] The clothing description input by the user is converted into a conditional vector through CLIP text encoding, and the cross-attention mechanism is embedded after the input layer. It is combined with the noise vector of the SDXL model as the input of the cross-attention mechanism, and the output of the cross-attention mechanism is used as the input of the backbone network of the SDXL model. The noise vector is guided by the conditional vector to guide the target image required by the user. The noise vector is the noise vector involved in the diffusion denoising of the SDXL model. In the cross-attention mechanism, the conditional vector is input into the VK linear layer, and the noise vector is input into the Q linear layer to ensure that the generated image meets the user's customization requirements.

[0058] Step 2: Model image generation Based on the input basic feature parameters, the system extracts the corresponding input data from the pre-built multi-dimensional model database to ensure the diversity and richness of the models. The input data is the vector or image data corresponding to the basic feature parameters, which is used as the input of the SDXL model to generate the final model image with the current input. The SDXL model also includes a noise vector in the input data. The noise vector is used as a latent variable. During the SDXL model generation process, the SDXL model is gradually guided to generate the corresponding model image through a step-by-step denoising method. The SDXL model is a diffusion model, and the input of the noise vector and the step-by-step denoising content are not described here.

[0059] The self-trained Stable Diffusion XL (SDXL) model is used to generate model images that meet the characteristics, ensuring that the model's appearance and posture are natural and realistic. The U-net backbone in the Stable Diffusion XL (SDXL) model is replaced with the DiT backbone; the Controlnet model is used to adjust the model's posture, where Controlnet uses residual connections to the attention mechanism, where the residual connection passes the input data in the attention mechanism to the output data of the attention mechanism, and combines the input data with the output data to generate a control embedding vector, which is injected into the cross-attention layer of the DiT backbone network to control the SDXL model's adjustment of the model's posture.

[0060] Step 3: Posture and movement adjustment The system uses the OpenPose model to estimate the pose of the generated model image and identify the key points of the body. After identifying the key points of the body, if they meet the pose parameters selected by the user, subsequent segmentation is performed. If they do not meet the requirements, the ControlNet model is used, and the body key points and the pose parameters selected by the user are used as the input of the ControlNet model, and the SDXL model is re-controlled to regenerate the model image.

[0061] According to the posture parameters selected by the user, the ControlNet model is used to accurately adjust the model's posture to ensure that the movement is natural and meets the design requirements, and the GroundingDINO and SAMSegment models are used to segment the image to form a model image mask. GroundingDINO is used to detect the position of the model in the relevant model image, and the SAMSegment model is used to segment the model area in the model image and form a model image mask.

[0062] In order to accurately adjust the model's posture, the ControlNet model uses residual connection and attention mechanism to inject the control embedding vector into the cross-attention layer of DiT, and controls the SDXL model based on the DiT network structure to regenerate the model image.

[0063] At the same time, by combining OpenPose, GroundingDINO and SAM Segment models, the model's posture is accurately estimated and adjusted to ensure that the posture is natural and meets design requirements.

[0064] Integration of posture adjustment algorithm: The posture adjustment algorithm is integrated into the model data processing flow to ensure that the posture of the generated model is highly consistent with user requirements.

[0065] Step 4: Clothing and accessories adjustment The system automatically generates or adjusts the model's clothing and accessories based on the clothing style and accessories selected by the user, ensuring a natural integration of the clothing and the model's posture.

[0066] For the integration between the above-mentioned clothing and the model, the above-mentioned clothing and the model’s posture are naturally fused through a generation method without additional reference network based on the Diffusion Transformer (DiT) to generate the model image: The present invention achieves seamless integration of conditional image inputs (including clothing images and masked portrait images, i.e., model image masks) by utilizing the inherent backbone of the diffuse transformer (DiT), avoids the use of additional reference networks or image encoders, significantly simplifies the model structure, and reduces computational complexity.

[0067] In the Transformer block of the diffusion transformer, the conditional image input is directly combined with the noisy image input to form a unified sequence input, thereby achieving collaborative processing of multimodal features.

[0068] Apply SDXL's Inpaint model to process the clothing's folds, texture, and light and shadow effects to ensure seamless integration of the clothing with the model's body.

[0069] For the seamless integration process, the present invention adopts high detail preservation and natural fusion technology: The seamless combination process includes the following: Combination of Stable Diffusion XL (SDXL) and Inpaint model: Through the SDXL model's high-resolution generation capability for model images and the Inpaint model's image synthesis function, the clothing and the model's body are seamlessly combined, and wrinkles, textures, and light and shadow effects are carefully processed to ensure the realism and coordination of the generated image.

[0070] Application location of the Inpaint model: As a model further connected to the SDXL model, the Inpaint model generates a model image and performs the above-mentioned posture detection, adjustment and segmentation. The segmented model image mask is further processed to refine the natural transition of the clothing itself and the connection details between the clothing and the model. It is applied to the image processing stage where the clothing is combined with the model's body. The clothing details are optimized through local redrawing technology to achieve natural fusion.

[0071] Step 5: Background fusion and optimization Based on the specific scene requirements of e-commerce display, the system automatically generates or selects a suitable background image.

[0072] The Inpaint model is used to naturally blend the generated model image with the background image, and the lighting, tone and perspective relationship are adjusted to ensure visual consistency and coordination.

[0073] Step 6: Multi-model collaboration and efficient parameter training The system uses G-LoRA and MP-LoRA to process subject generation and image restoration tasks respectively, and controls the LoRA status of different tasks through LoRA-switches to improve the flexibility and performance of the model.

[0074] Through LoRA technology, only a small number of trainable parameters are added to achieve efficient joint training, significantly reducing training time and memory consumption.

[0075] In the application of multi-task collaborative training and LoRA technology: Separate application of G-LoRA and MP-LoRA: The present invention introduces G-LoRA (for subject generation tasks) and MP-LoRA (for image restoration tasks) in the MM-Attention block of the Transformer in the Diffusion Transformer (DiT), and controls the LoRA states (disable, freeze, train) of different tasks through LoRA-switches to achieve efficient collaborative training of multiple tasks.

[0076] Control mechanism of LoRA-switches: Dynamically control the enabled status of G-LoRA and MP-LoRA through LoRA-switches to ensure that different tasks do not interfere with each other during joint training, thereby improving the flexibility and performance of the model.

[0077] Specific location of LoRA: G-LoRA and MP-LoRA are added to the QKV (query, key, value) linear layer in the MM-Attention block respectively to achieve efficient adaptation and fusion of different task features.

[0078] The core module of MM-Attention contains three linear transformation layers: Q, K, and V. G-LoRA and MP-LoRA introduce task-related features in the QKV layer through low-rank matrix decomposition: the weight matrix WQ / K / V of the original QKV linear layer is decomposed into: ,in ,and Among them, the A and B matrices are initialized to matrices that conform to Gaussian distribution and 0 respectively, and are optimized through training.

[0079] G-LoRA: acts on Q and K layers, , optimize the subject consistency of the generation task through global semantic alignment. For example: in text-to-image generation, by adjusting the QK interaction relationship, the matching of the text condition and the image latent vector is enhanced.

[0080] MP-LoRA: Prioritize injection into V layer, rank , improve the repair details (such as edge sharpening and texture restoration) through local feature reconstruction. For example: in image restoration, by dynamically adjusting the V layer weights, the pixel distribution of the mask area can be accurately reconstructed.

[0081] Implementation of LoRA-switches: The LoRA-switches mechanism is introduced to achieve precise control of G-LoRA and MP-LoRA by setting the enabled state of LoRA (disable, freeze, train), ensuring the stability and efficiency of multi-task training.

[0082] Disable: Completely turn off the LoRA module and restore the weight matrix to the original parameters ( ), does not participate in forward calculation and gradient update.

[0083] Freeze: Keep the weight of the LoRA module ( ), but with its parameters fixed for inference only.

[0084] Training: Activate the LoRA module and update its low-rank parameters (A and B), while freezing the main model weights (W).

[0085] Parameter efficiency vs. computation efficiency Low-Rank Adaptation (LoRA) technology: LoRA technology is used for parameter-efficient joint training, adding only a small number of trainable parameters (8 dimensions for G-LoRA and 4 dimensions for MP-LoRA), significantly reducing the computational cost of model training and inference.

[0086] Parameter sharing and optimization: Through the parameter sharing mechanism, the parameter utilization efficiency between different tasks is optimized, and the performance and computing efficiency of the overall model are improved.

[0087] Specific application of LoRA: G-LoRA and MP-LoRA are respectively added to the QKV linear layer in the MM-Attention block to optimize the adaptation and fusion of different task features.

[0088] Parameter sharing and control mechanism: LoRA-switches are used to control the LoRA status (disable, freeze, train) of different tasks, to achieve dynamic sharing and optimization of parameters and ensure efficient operation of the model under multi-task collaboration.

[0089] Step 7: Preview and fine-tune the generated results The designer previewed the generated model image in real time in the Photoshop interface and found that the model's hairstyle was slightly unnatural.

[0090] By adjusting clothing parameters (such as clothing style and clothing type), the system regenerates the model image that conforms to the adjustments to ensure a satisfactory final effect.

[0091] Step 8: Finalize the design The generated model image meets the satisfactory effect, which is applied to the e-commerce product display page to complete the design process.

[0092] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for automatically generating an AI model, characterized in that: include: Obtain the basic features, posture information and clothing description of the AI ​​model; Generate a model image according to basic features, clothing description and posture information through a diffusion transformer, and optimize the model image through an optimizer to obtain an optimized model image; wherein the diffusion transformer is optimized by combining LoRA; The optimized model image is subjected to background fusion and adjustment to obtain the final model image of the AI ​​model; The generation process of the model image includes: extracting input data from a model database according to the basic features, wherein the model database stores input data associated with the basic features; The input data and the clothing description are processed by the SDXL model to obtain an initial model image; wherein, in the SDXL model, the U-net backbone network structure is replaced by a diffusion transformer, and the clothing description is input by a cross attention mechanism, wherein the output of the cross attention mechanism is combined with the input data as the input of the diffusion transformer; the diffusion transformer is a DIT model; Performing pose estimation on the initial model image through the OpenPose model to obtain body key points; Adjusting the initial model image according to body key points and posture information using a ControlNet model, wherein the ControlNet model uses a residual connection attention mechanism to connect with a cross attention layer in a diffusion transformer in the SDXL model; The adjusted model image is segmented by the SAMSegment model to obtain the model image; The process of optimizing the diffusion converter by combining LoRA includes: G-LoRA and MP-LoRA are introduced into the MM-Attention block in the Transformer block in the diffusion transformer, and the states of G-LoRA and MP-LoRA are controlled by LoRA-switches, wherein the states include disabled, frozen, and trained; The G-LoRA and MP-LoRA are respectively added to the QKV linear layer in the MM-Attention block of the diffusion transformer, the G-LoRA is used to adjust the weights of the Q linear layer and the K linear layer for the subject generation task, and the MP-LoRA is used to adjust the weight of the V linear layer for the image restoration task; Among them, disable is used to completely shut down the LoRA module so that G-LoRA and MP-LoRA do not participate in forward calculation and gradient update; Freeze is used to keep the weights of the LoRA module but fix its parameters for inference only; Training is used to activate the LoRA modules and update their low-rank parameters while freezing the main model weights.

2. The method according to claim 1, characterized in that: Obtaining basic features, posture information, and clothing description of the AI ​​model through an interactive interface, wherein the interactive interface includes input prompts and input methods for the basic features, posture information, and clothing description of the AI ​​model, and the interactive interface is also used to interactively display the model image, clothing image, fused image, and final image; The basic characteristics include gender, age, skin color and expression; the clothing description includes clothing style and accessories information.

3. The method according to claim 1, characterized in that Before generating the model image, it also includes; The clothing description is encoded and transformed through CLIP text encoding to obtain a conditional vector, and the conditional vector and the noise vector of the SDXL model are used as inputs of a cross-attention mechanism. The output of the cross-attention mechanism is combined with input data as inputs of a diffusion transformer.

4. The method according to claim 1, characterized in that: The process of generating the optimized model image includes: The model image is adjusted in detail using the Inpaint model to obtain an optimized model image.

5. The method according to claim 1, characterized in that The process of background fusion and adjustment of the fused image includes: A background image is obtained, and a fused image is fused according to the background image to obtain a background fused image. The background fused image is adjusted to obtain a final model image of the AI ​​model, wherein the adjustment includes adjustment of lighting, color tone, and perspective relationship.

6. The method according to claim 1, characterized in that The final model image is generated and includes: If the final model image does not meet the requirements, the basic features, posture information and clothing description of the AI ​​model are readjusted, and the final model image is generated based on the readjusted data.

Images