Image generation method and device and storage medium

Abstract

The embodiment of the invention provides an image generation method and device and a storage medium, and belongs to the field of artificial intelligence graphics, and the method comprises the steps that text description and control conditions input by a user are acquired, and the text description comprises main object information, environment scene information, action interaction information and style preference information; cleaning, splitting and optimizing the text description to obtain a word segmentation result, and translating and aligning the word segmentation result to obtain key information of a Chinese semantic vector; constructing an image generation basic model according to the key information, the control condition and the text description; determining an adjustment task according to a user demand, determining a training data set of the LoRA model according to the adjustment task, and training the training data set to obtain a LoRA weight file; loading the LoRA weight file to the image generation basic model to obtain a preliminary image; and carrying out quality detection on the initial image to obtain a personalized image. According to the method, efficient and high-quality image generation is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence text-to-image generation, specifically to an image generation method, apparatus, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, text-to-image (TTO) technology has become an important research direction in the field of image generation. Most existing TTO platforms rely on a single model, lacking flexibility across different datasets or application scenarios. Furthermore, their training methods, data, and generation methods vary, making it difficult to simultaneously meet the demands for generation efficiency, image quality, and controllability. Moreover, due to the diverse needs of users, a single model cannot achieve high-quality full-scale matching for different application scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide an image generation method, apparatus, and storage medium, which achieves efficient and high-quality image generation.

[0004] To achieve the above objectives, embodiments of the present invention provide an image generation method, the method comprising: The system obtains text descriptions and control conditions input by the user, wherein the text descriptions include subject object information, environmental scene information, action interaction information, and style preference information; The text description is cleaned, split, and optimized to obtain word segmentation results. The word segmentation results are then translated and aligned to obtain key information of the Chinese semantic vector. Construct a basic model for image generation based on the key information, control conditions, and text description; The adjustment task is determined according to the user's needs, the training dataset of the LoRA model is determined according to the adjustment task, and the LoRA weight file is obtained by training the training dataset. The LoRA weight file is loaded into the image generation base model to obtain a preliminary image; A personalized image is obtained by performing quality inspection on the preliminary image.

[0005] Optionally, constructing the image generation base model based on the key information, control conditions, and text description includes: Based on the style preference information and control conditions of the key information, an appropriate image generation base model is determined, and the pre-training weights of the image generation base model are determined based on the key information for training the image generation base model. The text description is mapped to a text feature vector by a text encoder, which is used for semantic and visual feature space alignment in the image generation base model. The attention layer parameters of the image generation base model are adjusted according to the weight allocation of the key information; The main body region and background region of the image generation basic model are determined based on the positional relationship of the main object information, environmental scene information, and action interaction information.

[0006] Optionally, the method further includes: optimizing and training the image generation base model using the key information, including: A sample fine-tuning strategy is adopted, using reference images corresponding to key information as training data, freezing the backbone network of the image generation basic model, and training the alignment layer of the image generation basic model. The generated preliminary image and reference image are similar to each other and iterated adversarially based on the adversarial training mechanism and discriminator network to optimize the basic image generation model.

[0007] Optionally, the step of determining the adjustment task based on user needs, determining the training dataset for the LoRA model based on the adjustment task, and training the LoRA weight file on the training dataset includes: The adjustment tasks are determined based on user needs, including color adjustment tasks, texture addition tasks, and style optimization tasks. A LoRA adapter is inserted into the key layer of the image generation base model using low-rank matrix factorization. Determine the LoRA model parameters and training hyperparameters based on the aforementioned adjustment task; A training dataset is constructed based on the LoRA model parameters and training hyperparameters. The LoRA weight file is obtained by training the training dataset using a freeze-fine-tuning strategy and feedback iterative training.

[0008] Optionally, the method further includes: obtaining user feedback information; The diffusion process parameters of the image generation base model and the number of iterations of the super-resolution network are adjusted based on the feedback information. The diffusion process parameters include the number of diffusion steps and a noise scheduler. The parameters and weights of the LoRA model are adjusted based on the feedback information.

[0009] Optionally, the method further includes adding additional conditions through the ControlNet model during the generation of personalized images, the additional conditions including color, style, and layout.

[0010] Optionally, the control conditions include: resolution parameters, generation speed constraints, accuracy requirements, content compliance constraints, and format output requirements.

[0011] On the other hand, the present invention also proposes an image generation apparatus, which includes: The acquisition module is used to acquire the text description and control conditions input by the user. The text description includes subject object information, environmental scene information, action interaction information and style preference information. The first processing module is used to clean, split, and optimize the text description to obtain word segmentation results, and translate and align the word segmentation results to obtain key information of the Chinese semantic vector. The second processing module is used to construct a basic model for image generation based on the key information, control conditions and text description. The third processing module is used to determine the adjustment task according to the user's needs, determine the training dataset of the LoRA model according to the adjustment task, and train the training dataset to obtain the LoRA weight file. The fourth processing module is used to load the LoRA weight file into the image generation base model to obtain a preliminary image; The fifth processing module is used to perform quality detection on the preliminary image to obtain a personalized image.

[0012] Optionally, constructing the image generation base model based on the key information, control conditions, and text description includes: Based on the style preference information and control conditions of the key information, an appropriate image generation base model is determined, and the pre-training weights of the image generation base model are determined based on the key information for training the image generation base model. The text description is mapped to a text feature vector by a text encoder, which is used for semantic and visual feature space alignment in the image generation base model. The attention layer parameters of the image generation base model are adjusted according to the weight allocation of the key information; The main body region and background region of the image generation basic model are determined based on the positional relationship of the main object information, environmental scene information, and action interaction information.

[0013] On the other hand, the present invention also proposes a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the image generation method described above.

[0014] An image generation method of the present invention includes: acquiring a text description and control conditions input by a user, wherein the text description includes subject object information, environmental scene information, action interaction information, and style preference information; cleaning, splitting, and optimizing the text description to obtain word segmentation results, and translating and aligning the word segmentation results to obtain key information of Chinese semantic vectors; constructing a basic image generation model based on the key information, control conditions, and text description; determining an adjustment task according to user needs, determining a training dataset for a LoRA model based on the adjustment task, and training the training dataset to obtain a LoRA weight file; loading the LoRA weight file into the basic image generation model to obtain a preliminary image; and performing quality detection on the preliminary image to obtain a personalized image. This method integrates a large language model, a basic image generation model, a LoRA model, and a ControlNet model to achieve efficient and high-quality image generation, and allows users to finely control the generation process by adding additional conditions, thus offering greater flexibility, controllability, and practicality.

[0015] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating an image generation method according to the present invention; Figure 2 This is a schematic diagram of one embodiment of the present invention; Figure 3 This is a schematic diagram of an image generation device according to the present invention.

[0017] Explanation of reference numerals in the attached figures 100 - Image generation device; 200 - Acquisition module; 300 - First processing module; 400 - Second processing module; 500 - Third processing module; 600 - Fourth processing module; 700 - Fifth processing module. Detailed Implementation

[0018] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0019] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0020] Example 1 This invention provides an image generation method, such as... Figure 1 As shown, the method includes: Step S101 involves obtaining the text description and control conditions input by the user. The text description includes information about the main object, the environment and scene, the action interaction, and the style preference. The control conditions include: resolution parameters, generation speed constraints, accuracy requirements, content compliance constraints, and format output requirements.

[0021] In one specific implementation, the subject object information is the core entity to be presented in the image, such as "a white Persian cat" or "a retro wooden desk," which includes entity category (animal / furniture, etc.), appearance characteristics (color, material, shape), and quantity attributes (single / multiple).

[0022] The environmental scene information refers to the space and atmosphere in which the subject is located, such as "a beach at sunset" or "a laboratory at night," which includes scene type (natural / indoor, etc.), time dimension (early morning / night), lighting conditions (strong light / soft light), and weather conditions (sunny / rainy).

[0023] The action interaction information describes the dynamic behavior of the subject or the relationship between entities, such as "a Persian cat lying on the desk licking its paws" or "an experimenter stirring liquid with a beaker in hand". It is necessary to specify the action type (static posture / dynamic behavior), the interaction object and the interaction method.

[0024] The style preference information refers to the artistic expression of the specified image, such as "Impressionist oil painting style", "Japanese anime style", and "realistic photography style", which includes style category, brushstroke characteristics (delicate / rough), color tone (bright / dark), etc.

[0025] The resolution parameter sets the pixel size of the output image, such as "1024×1024 pixels" or "2048×1536 pixels," and supports custom aspect ratios (e.g., 16:9, 4:3). The generation speed constraint limits the image generation time based on the application scenario, such as "real-time generation on mobile devices (≤1 second)" or "high-precision generation for professional design scenarios (≤10 seconds)." The precision requirement defines the level of image detail, such as "low precision (only retaining the main outline)", "high precision (representing hair texture / wood grain details)", or "ultra-precision (restoring the reflection / shadow levels of objects)."

[0026] The content compliance constraints exclude the generation of inappropriate content, such as "prohibiting the generation of inappropriate elements" and "filtering inappropriate images," and support custom sensitive content blocking. The output format requirements specify the image file format, such as JPEG, PNG, and TIFF, and support transparent backgrounds (PNG / TIFF format only) and compression quality parameters (such as JPEG compression ratio of 80%-100%).

[0027] Step S102 involves cleaning, splitting, and optimizing the text description to obtain word segmentation results, and then translating and aligning the word segmentation results to obtain key information of the Chinese semantic vector.

[0028] According to one specific implementation method, the text description input by the user is cleaned to remove redundant characters (such as special symbols and repeated words) and correct grammatical errors (such as improper subject-verb-object collocation); the text is split into semantic units using word segmentation tools (such as BPE and WordPiece), and the word segmentation results are optimized by combining a domain dictionary (a dedicated vocabulary library for text-based graphics, containing professional terms such as "brushstroke" and "light and shadow"); the multilingual text description is uniformly translated and aligned, and non-Chinese text is converted into Chinese semantic vectors to ensure consistent understanding of cross-language descriptions.

[0029] The semantic understanding module of a large language model (such as the GPT series or LLaMA series) is invoked, and the model outputs structured information through a Prompt project. An example Prompt template is: "Extract four types of information from the following text description: subject object, environment scene, action interaction, and style preference, and output in JSON format: [User Input Text]". The model output is then subjected to entity recognition and classification. Named Entity Recognition (NER) technology is used to label the subject object (e.g., "Persian cat" is labeled as "animal-pet") and the environment scene (e.g., "beach" is labeled as "natural scene-water"). Relationship extraction algorithms are used to identify relationships between entities (e.g., the action relationship "Persian cat-lying-on-desk"). The large language model is a complete system that processes raw materials and provides retrieval capabilities to the business, including modules for material storage, slicing, vectorization, index building, and online retrieval.

[0030] For ambiguous descriptions (such as "beautiful flowers"), semantic completion is performed. Based on the knowledge graph of the large language model, specific features corresponding to "beautiful" are added (such as "rich petal layers and bright colors"). If there are polysemous descriptions (such as "apple" may refer to fruit or brand), the ambiguity is eliminated through contextual semantics (such as "apple placed on a plate" is determined to be fruit).

[0031] Based on the importance of each semantic unit in the text description, weights are extracted through the attention mechanism of the large language model. For example, in "The red sports car is speeding on the track", the weight of "red sports car" (subject) is set to 0.4, the weight of "track" (environment) is set to 0.3, the weight of "speeding" (action) is set to 0.2, and the weight of "red" (appearance) is set to 0.1.

[0032] Based on user's historical generation records, the weight of frequently occurring semantic units (such as users frequently generating "Japanese anime style" images) is increased, prioritizing the retention of core preference information; structured key information is output, with the following format example: json{"Main Object": [{"Name": "Red Sports Car", "Attribute": {"Color": "Red", "Type": "Transportation - Car"},"Weight": 0.4}],"Environment Scene": [{"Name": "Racing Track", "Attribute": {"Type": "Artificial Scene - Transportation Scene"}, "Weight": 0.3}],"Action Interaction": [{"Action": "Speeding", "Main": "Red Sports Car", "Weight": 0.2}],"Style Preference": [{"Style": "Realistic Style", "Attribute": {"Level of Detail": "High Precision"},"Weight": 0.1}]}.

[0033] Step S103 involves constructing a basic image generation model based on the key information, control conditions, and text description. This basic image generation model is a text-to-image model, used for facilitating retrieval of full-text indexes and question-and-answer pairs in question-and-answer databases after processing the original material.

[0034] According to one specific implementation, constructing an image generation basic model based on the key information, control conditions, and text description includes: determining an adapted image generation basic model based on the style preference information and control conditions of the key information; determining the pre-training weights of the image generation basic model based on the key information for training the image generation basic model; mapping the text description into text feature vectors through a text encoder for semantic and visual feature space alignment in the image generation basic model; adjusting the attention layer parameters of the image generation basic model based on the weight allocation of the key information; and determining the subject region and background region of the image generation basic model based on the positional relationship of the subject object information, environmental scene information, and action interaction information.

[0035] The method further includes: optimizing the training of the image generation basic model using the key information, including: adopting a sample fine-tuning strategy, using the reference image corresponding to the key information as training data, freezing the backbone network of the image generation basic model, and training the alignment layer of the image generation basic model; performing similarity judgment and adversarial iteration on the generated preliminary image and the reference image according to the adversarial training mechanism and the discriminator network, in order to optimize the image generation basic model.

[0036] Specifically, this method includes the selection and initialization of the basic image generation model, mapping key information to the model network, and model training and optimization. The selection and initialization of the basic image generation model involves choosing a suitable basic image generation model architecture based on style preferences and detail accuracy requirements found in the key information. For example, the Stable Diffusion XL model is used for realistic styles and high-precision scenes, the Anything model is used for anime-style scenes, and the MobileDiffusion model is used for low-precision, fast-generation scenes.

[0037] Load the pre-trained weights of the model, and import the corresponding pre-trained weights (such as fine-tuned weights for mechanical design) for the domain features in the key information (such as "industrial machinery design drawings"), and initialize the model's image generation network (including encoder, decoder, and diffusion process module).

[0038] The mapping of key information to the model network includes: converting structured key information into feature vectors that the model can recognize; mapping semantic information such as "subject object" and "environment scene" into 768-dimensional / 1024-dimensional text feature vectors through a text encoder (such as CLIP text encoder) to ensure alignment between semantic and visual feature spaces. For the weight allocation in the key information, the parameters of the model's attention layer are adjusted: for high-weight semantic units (such as "red sports car"), the matching weight between their text features and image features is increased in the cross-attention module, increasing the visual proportion of this entity in the generated image.

[0039] A scene layout generation submodule is constructed to generate an initial image layout heatmap based on the positional relationship between the "environment scene" and the "main object" (such as "the sports car is in the center of the track"). The main area (high heat value in the center of the track) and the background area (low heat value at the edge of the track) are defined to provide spatial constraints for the subsequent diffusion process.

[0040] The model training and optimization include: employing a few-shot fine-tuning strategy, using reference images corresponding to key information (such as a standard image library of "red sports cars") as training data, freezing the model's backbone network, and training only the text-image feature alignment layer to enable the model to more accurately capture key semantics; introducing an adversarial training mechanism to construct a discriminator network, which judges the similarity between the generated initial image and the reference image, and optimizing the model's generation details (such as the streamlined body and tire texture of the sports car) through adversarial iteration between the generator and the discriminator; for model generation efficiency, employing model quantization (such as INT8 quantization) and pruning techniques to reduce the model parameter size (such as reducing the StableDiffusion XL model parameters from 13 billion to 8 billion) while ensuring generation quality, thus meeting the generation speed requirements in the control conditions; and outputting a basic image generation model, which includes initial weights, feature mapping rules, and layout constraint parameters, capable of generating initial images that meet basic semantic and scene requirements based on the input key information.

[0041] Step S104 involves determining the adjustment task based on user needs, determining the training dataset for the LoRA model based on the adjustment task, and training the LoRA weight file using the training dataset.

[0042] LoRA (Retrieval Augmented Generation) combines information retrieval with a carefully crafted set of system prompts, anchoring LLM to accurate, up-to-date, and relevant information retrieved from external knowledge stores.

[0043] According to one specific implementation, the steps of determining adjustment tasks based on user needs, determining a training dataset for the LoRA model based on the adjustment tasks, and training the training dataset to obtain a LoRA weight file include: determining adjustment tasks based on user needs, the adjustment tasks including color adjustment tasks, texture addition tasks, and style optimization tasks; inserting a LoRA adapter into the key layers of the image generation base model using low-rank matrix factorization; determining LoRA model parameters and training hyperparameters based on the adjustment tasks; constructing a training dataset based on the LoRA model parameters and training hyperparameters; and training the training dataset using a freeze-fine-tuning strategy and feedback iterative training to obtain a LoRA weight file.

[0044] Specifically, user needs analysis and LoRA tasks include: collecting personalized user needs, divided into explicit needs (such as "changing the red sports car to blue, adding carbon fiber texture") and implicit needs (such as "enhancing the technological feel of the image, adjusting the lighting to side lighting"), obtaining detailed needs through questionnaires or interactive interfaces; mapping user needs to LoRA model adjustment tasks, clarifying adjustment goals: such as color adjustment tasks (modifying RGB channel parameters), texture addition tasks (integrating texture feature layers), and style optimization tasks (updating style feature vectors).

[0045] Based on the priority of needs, the adjustment tasks are sorted. For example, "color modification" (high priority, directly affecting the appearance of the main body) takes precedence over "light and shadow adjustment" (medium priority, affecting the visual atmosphere), and task weights are set for LoRA model training.

[0046] The construction and parameter configuration of the LoRA model include: designing the LoRA model structure, using low-rank matrix factorization, and inserting LoRA adapters into key layers of the image generation base model (such as the text encoder output layer and the ResNet layer of the diffusion process). Each adapter contains an A matrix (dimension: d_model × r) and a B matrix (dimension: r × d_model), where r is the rank parameter (usually set to 4-64, adjusted according to the required complexity, such as r=32 for texture addition tasks and r=16 for color adjustment tasks); initializing the LoRA model parameters, with the A matrix initialized using a random normal distribution and the B matrix initialized to zero, ensuring that the LoRA model does not affect the output of the image generation base model in the initial state, and personalized features are only gradually injected through subsequent training.

[0047] Configure LoRA training hyperparameters: set the learning rate to 1e-4-5e-4 (adjust according to task complexity, take a higher learning rate for complex texture tasks), set the batch size to 4-8 (balance training efficiency and memory usage), and set the training epochs to 50-200 (based on the required convergence, such as color adjustment tasks can converge in 50 epochs).

[0048] The LoRA model training and fine-tuning includes: constructing a personalized training dataset: collecting a small number of reference samples (such as 5-20 reference images of a "blue carbon fiber sports car") based on user needs, preprocessing the samples (e.g., normalizing the size to 512×512 pixels, data augmentation: flipping, rotating); adopting a freeze-tuning strategy: freezing all parameters of the image generation base model, training only the A and B matrices of the LoRA adapter, calculating the loss through backpropagation (using the MSE loss function to calculate the pixel-level error between the generated image and the reference sample, combined with the perceptual loss function to improve visual similarity); introducing user feedback for iterative training: every 20 training rounds, generating intermediate personalized images and displaying them to the user; if the user provides feedback (e.g., "the carbon fiber texture is not clear enough"), adjusting the training samples (adding high-resolution texture samples) and the loss function weights (increasing the loss weight of texture features), and retraining until the user's needs are met. After training, extracting the adapter parameters (A and B matrices) of the LoRA model, generating a lightweight LoRA weight file (typically tens to hundreds of MB in size, easy to store and retrieve).

[0049] Step S105 involves loading the LoRA weight file into the image generation base model to obtain a preliminary image.

[0050] According to one specific implementation method, the trained LoRA weight file is loaded into the image generation base model, and a preliminary personalized image is generated through the LoRA fusion strategy (such as weighted summation: image generation base model output + LoRA adapter output × weight coefficient, with the weight coefficient set to 0.7-1.0 to balance basic semantics and personalized features).

[0051] Step S106 involves performing quality checks on the preliminary image to obtain a personalized image.

[0052] According to a specific implementation method, the initial image is subjected to quality inspection, and the generated quality is judged by image evaluation indicators (such as FID, IS score). If there are defects in details (such as texture blurring, color deviation), the LoRA weight coefficient is adjusted (such as increasing the coefficient to 0.9 if the texture is blurry) or the LoRA model is fine-tuned. The final personalized image is output, and the LoRA adjustment parameters (task type, rank parameter, weight coefficient) are saved to the user configuration file for easy access in subsequent similar needs.

[0053] The method further includes: obtaining user feedback information; adjusting the diffusion process parameters of the image generation base model and the number of iterations of the super-resolution network according to the feedback information, wherein the diffusion process parameters include the number of diffusion steps and a noise scheduler; and adjusting the parameters and weights of the LoRA model according to the feedback information.

[0054] Specifically, based on user feedback and real-time generated personalized images, the parameters and weights of the large language model, the image generation basic model, and the LoRA model are dynamically adjusted. The specific techniques include: feedback acquisition and quantization processing, large language model parameter adjustment, image generation basic model parameter adjustment, and LoRA model weight adjustment.

[0055] The feedback collection and quantification process includes: constructing a multi-dimensional feedback collection system: subjective feedback: users submit opinions through ratings (1-5 points), tag selection (such as "color mismatch", "insufficient details"), and text descriptions (such as "hope for softer lighting"); objective feedback: the system automatically collects image quality data (such as resolution compliance rate, generation time, compliance test results) and user interaction behavior (such as image save rate, number of secondary modifications, and dwell time).

[0056] The feedback information is quantified, and subjective scores are mapped to quantitative indicators (e.g., 5 points = 1.0, 4 points = 0.8, and so on). Keyword extraction and value assignment are performed on text feedback through a large language model (e.g., "color mismatch" is assigned "color deviation = 0.6"). Objective data is normalized (e.g., the generation time is normalized from "5 seconds" to "time index = 0.5", based on the 10-second upper limit in the control conditions).

[0057] Establish a feedback weight model and set weights according to the feedback type. The weight of subjective feedback (0.6) > the weight of objective quality data (0.3) > the weight of interactive behavior (0.1). Calculate the comprehensive feedback score (e.g., comprehensive score = subjective score × 0.6 + quality compliance rate × 0.3 + retention rate × 0.1). If the score is < 0.6, it is determined that the model needs to be adjusted.

[0058] The large language model parameter adjustments include: if the feedback points to "semantic understanding bias" (e.g., the user description "retro desk" generates "modern desk"), adjust the semantic parsing module of the large language model: optimize the Prompt template and add domain constraint words, such as changing the original template "extract key information" to "extract the main object and environmental scene information in the text-to-image scene, focusing on retro and other style keywords".

[0059] Fine-tune the model's attention mechanism, increasing the attention weight for low-frequency keywords (such as "retro" and "carbon fiber"). Perform prompt tuning using a small number of feedback samples (such as 10-20 sets of "description-correct parsing" data) to update the model's semantic mapping rules. If the feedback points to "ambiguity handling errors" (such as "apple" being misclassified as a brand), update the knowledge graph of the large language model, supplement entity association data in text-to-image scenarios (such as "apple + plate → fruit" and "apple + logo → brand"), and adjust the threshold of the entity recognition algorithm (such as increasing the brand entity recognition threshold from 0.5 to 0.7 to reduce misclassification). After adjustment, test the model's parsing accuracy using new text description samples (including error-prone descriptions from historical feedback). The goal is to improve the accuracy by ≥10% from before the adjustment. If the target is not met, repeat the above adjustment steps.

[0060] Adjustments to the basic image generation model parameters include: If feedback indicates "inadequate generation quality" (e.g., blurred details, scene distortion), adjust the diffusion process parameters of the basic image generation model: increase the number of diffusion steps (e.g., from 50 steps to 100 steps) to extend the detail optimization time during image generation; adjust the noise scheduler (e.g., change the DDIM scheduler to the DPM++2MKarras scheduler) to improve the stability of image texture and color; fine-tune the super-resolution module of the model; if user feedback indicates "insufficient resolution," increase the number of iterations of the super-resolution network (e.g., from 2 to 3 times) to optimize the image pixel filling algorithm; if feedback indicates "low generation efficiency" (e.g., time consumption exceeds control requirements), perform lightweight optimization of the basic image generation model while ensuring quality.

[0061] Model pruning techniques are employed to remove redundant convolutional layers (e.g., pruning ResNet blocks with a contribution of less than 5%); model accuracy is reduced (e.g., quantization from FP32 to FP16) to reduce memory usage and computation time; after adjustments, quality and efficiency are verified through batch testing (generating 100 images of different scenes), requiring an improvement of ≥0.1 in the overall feedback score while reducing generation time by ≥20%.

[0062] LoRA model weight adjustments include: If feedback indicates "insufficient personalized features" (e.g., a "blue sports car" still appears purplish), adjust the LoRA model's weight coefficients: increase the weight of the corresponding adjustment task (e.g., increase the weight of the color adjustment task from 0.7 to 0.9); increase the rank parameter of the LoRA model (e.g., from r=16 to r=32) to enhance the model's ability to capture detailed features; If feedback indicates "excessive personalized features" (e.g., "carbon fiber texture" obscures the car's outline), adopt a weight decay strategy: decrease the LoRA weight coefficients (e.g., from 0.9 to 0.6); apply a threshold limit to the LoRA adapter's output (e.g., ...). To avoid excessive stacking, limit the output value of texture features to the range of 0-0.8. If the feedback indicates "poor adaptation to similar needs" (e.g., inconsistent generation results for "tech-style" needs among different users), construct a general LoRA fine-tuning dataset: collect 50-100 reference images of "tech-style" from different scenes; fine-tune the LoRA model a second time based on this dataset to optimize the model's generalization ability; after adjustment, ensure that the personalized feature conformity of the generated images is improved by ≥30% through 3-5 tests of similar needs by the same user, and that the consistency of generation results for similar needs among different users (using the PSNR index) is ≥85%.

[0063] The method further includes adding additional conditions through the ControlNet model during the generation of personalized images. These additional conditions include color, style, and layout. ControlNet is used for raw data files such as Word, PDF, PPT, TXT, Excel, database tables, multimedia data, and web page data awaiting processing.

[0064] This application utilizes a multi-model coordinated image generation platform. By integrating a large language model, a basic image generation model, a LoRA model, and a ControlNet model, it achieves efficient and high-quality image generation and allows users to finely control the generation process by adding additional conditions. Compared to traditional image generation platforms, this application offers greater flexibility, controllability, and practicality, and can be widely applied in various fields such as artistic creation, advertising design, and product design.

[0065] Example 2 like Figure 2 As shown, this invention proposes an implementation of a system architecture, which includes an input module, a text-based image basic model generation module, a LoRA model fine-tuning module, a ControlNet control module, a parameter control calling model, and an output module.

[0066] The input module is used to receive text descriptions and optional additional control conditions input by the user.

[0067] The large language model processing module is used to understand and analyze the input text description using a large language model, and extract key information.

[0068] The text-to-image basic model generation module is used to generate preliminary images based on the text information processed by the large language model and the text-to-image basic model.

[0069] The LoRA model fine-tuning module is used to fine-tune the basic model of the text image based on user needs, thereby achieving personalized image generation.

[0070] The ControlNet control module is used to add additional conditions, such as color, style, and layout, through the ControlNet model during the image generation process, so as to perform fine control over image generation.

[0071] The parameter control calling model can be used by experts to select portrait models, traditional Chinese style models, etc., making full use of the model's characteristics to achieve specialized models for specific purposes.

[0072] The output module is used to output the final generated image and provide it to the user for download or further editing.

[0073] The modules transmit data through standardized data interfaces to ensure the accuracy and completeness of information. At different stages of image generation, appropriate models are selected and fused according to actual needs to fully utilize the advantages of each model. Based on user feedback and the real-time image quality, the parameters and weights of each model are dynamically adjusted to achieve the best generation results.

[0074] This system architecture integrates a large language model, a text-to-image basic model, a LoRA model, and a ControlNet model to achieve efficient and high-quality image generation, and allows users to finely control the generation process by adding additional conditions.

[0075] Example 3 The present invention also proposes an image generation apparatus, such as... Figure 3As shown, the image generation device 100 includes: an acquisition module 200, used to acquire text descriptions and control conditions input by a user, wherein the text descriptions include subject object information, environmental scene information, action interaction information, and style preference information; a first processing module 300, used to clean, split, and optimize the text descriptions to obtain word segmentation results, and to translate and align the word segmentation results to obtain key information of Chinese semantic vectors; a second processing module 400, used to construct an image generation basic model based on the key information, control conditions, and text descriptions; a third processing module 500, used to determine adjustment tasks according to user needs, determine the training dataset of the LoRA model based on the adjustment tasks, and train the training dataset to obtain a LoRA weight file; a fourth processing module 600, used to load the LoRA weight file into the image generation basic model to obtain a preliminary image; and a fifth processing module 700, used to perform quality detection on the preliminary image to obtain a personalized image.

[0076] According to one specific implementation, constructing an image generation base model based on the key information, control conditions, and text description includes: determining an adapted image generation base model based on the style preference information and control conditions of the key information; determining pre-training weights of the image generation base model based on the key information for training the image generation base model; mapping the text description into text feature vectors using a text encoder for semantic and visual feature space alignment in the image generation base model; adjusting the attention layer parameters of the image generation base model based on the weight allocation of the key information; and determining the subject region and background region of the image generation base model based on the positional relationship of the subject object information, environmental scene information, and action interaction information. This device achieves efficient and high-quality image generation.

[0077] An image generation method of the present invention includes: acquiring a text description and control conditions input by a user, wherein the text description includes subject object information, environmental scene information, action interaction information, and style preference information; cleaning, splitting, and optimizing the text description to obtain word segmentation results, and translating and aligning the word segmentation results to obtain key information of Chinese semantic vectors; constructing a basic image generation model based on the key information, control conditions, and text description; determining an adjustment task according to user needs, determining a training dataset for a LoRA model based on the adjustment task, and training the training dataset to obtain a LoRA weight file; loading the LoRA weight file into the basic image generation model to obtain a preliminary image; and performing quality detection on the preliminary image to obtain a personalized image. This method integrates a large language model, a basic image generation model, a LoRA model, and a ControlNet model to achieve efficient and high-quality image generation, and allows users to finely control the generation process by adding additional conditions, thus offering greater flexibility, controllability, and practicality.

[0078] This invention provides a storage medium storing a program that, when executed by a processor, implements the image generation method.

[0079] This invention provides a processor for running a program, wherein the program executes the image generation method during runtime.

[0080] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring a text description and control conditions input by a user, wherein the text description includes subject object information, environmental scene information, action interaction information, and style preference information; cleaning, splitting, and optimizing the text description to obtain word segmentation results, and translating and aligning the word segmentation results to obtain key information of the Chinese semantic vector; constructing a basic image generation model based on the key information, control conditions, and text description; determining an adjustment task based on user needs, determining a training dataset for the LoRA model based on the adjustment task, and training the training dataset to obtain a LoRA weight file; loading the LoRA weight file into the basic image generation model to obtain a preliminary image; and performing quality detection on the preliminary image to obtain a personalized image. The device described herein can be a server, PC, PAD, mobile phone, etc.

[0081] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: acquiring a text description and control conditions input by a user, wherein the text description includes subject object information, environmental scene information, action interaction information, and style preference information; cleaning, splitting, and optimizing the text description to obtain word segmentation results, and translating and aligning the word segmentation results to obtain key information of Chinese semantic vectors; constructing an image generation basic model based on the key information, control conditions, and text description; determining an adjustment task based on user needs, determining a training dataset for the LoRA model based on the adjustment task, training the training dataset to obtain a LoRA weight file; loading the LoRA weight file into the image generation basic model to obtain a preliminary image; and performing quality detection on the preliminary image to obtain a personalized image.

[0082] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0087] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0088] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0090] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image generation method, characterized in that, The method includes: The system obtains text descriptions and control conditions input by the user, wherein the text descriptions include subject object information, environmental scene information, action interaction information, and style preference information; The text description is cleaned, split, and optimized to obtain word segmentation results. The word segmentation results are then translated and aligned to obtain key information of the Chinese semantic vector. Construct a basic model for image generation based on the key information, control conditions, and text description; The adjustment task is determined according to the user's needs, the training dataset of the LoRA model is determined according to the adjustment task, and the LoRA weight file is obtained by training the training dataset. The LoRA weight file is loaded into the image generation base model to obtain a preliminary image; A personalized image is obtained by performing quality inspection on the preliminary image.

2. The method according to claim 1, characterized in that, The step of constructing the basic image generation model based on the key information, control conditions, and text description includes: Based on the style preference information and control conditions of the key information, an appropriate image generation base model is determined, and the pre-training weights of the image generation base model are determined based on the key information for training the image generation base model. The text description is mapped to a text feature vector by a text encoder, which is used for semantic and visual feature space alignment in the image generation base model. The attention layer parameters of the image generation base model are adjusted according to the weight allocation of the key information; The main body region and background region of the image generation basic model are determined based on the positional relationship of the main object information, environmental scene information, and action interaction information.

3. The method according to claim 1, characterized in that, The method further includes: optimizing and training the image generation base model using the key information, including: A sample fine-tuning strategy is adopted, using reference images corresponding to key information as training data, freezing the backbone network of the image generation basic model, and training the alignment layer of the image generation basic model. The generated preliminary image and reference image are similar to each other and iterated adversarially based on the adversarial training mechanism and discriminator network to optimize the basic image generation model.

4. The method according to claim 1, characterized in that, The process of determining the adjustment task based on user needs, determining the training dataset for the LoRA model based on the adjustment task, and training the LoRA weight file on the training dataset includes: The adjustment tasks are determined based on user needs, including color adjustment tasks, texture addition tasks, and style optimization tasks. A LoRA adapter is inserted into the key layer of the image generation base model using low-rank matrix factorization. Determine the LoRA model parameters and training hyperparameters based on the aforementioned adjustment task; A training dataset is constructed based on the LoRA model parameters and training hyperparameters. The LoRA weight file is obtained by training the training dataset using a freeze-fine-tuning strategy and feedback iterative training.

5. The method according to claim 1, characterized in that, The method also includes: Obtain user feedback information; The diffusion process parameters of the image generation base model and the number of iterations of the super-resolution network are adjusted based on the feedback information. The diffusion process parameters include the number of diffusion steps and a noise scheduler. The parameters and weights of the LoRA model are adjusted based on the feedback information.

6. The method according to claim 1, characterized in that, The method also includes: In the process of generating personalized images, additional conditions are added through the ControlNet model, including color, style, and layout.

7. The method according to claim 1, characterized in that, The control conditions include: resolution parameters, generation speed constraints, accuracy requirements, content compliance constraints, and format output requirements.

8. An image generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire the text description and control conditions input by the user. The text description includes subject object information, environmental scene information, action interaction information and style preference information. The first processing module is used to clean, split, and optimize the text description to obtain word segmentation results, and translate and align the word segmentation results to obtain key information of the Chinese semantic vector. The second processing module is used to construct a basic model for image generation based on the key information, control conditions and text description. The third processing module is used to determine the adjustment task according to the user's needs, determine the training dataset of the LoRA model according to the adjustment task, and train the training dataset to obtain the LoRA weight file. The fourth processing module is used to load the LoRA weight file into the image generation base model to obtain a preliminary image; The fifth processing module is used to perform quality detection on the preliminary image to obtain a personalized image.

9. The apparatus according to claim 8, characterized in that, The step of constructing the basic image generation model based on the key information, control conditions, and text description includes: Based on the style preference information and control conditions of the key information, an appropriate image generation base model is determined, and the pre-training weights of the image generation base model are determined based on the key information for training the image generation base model. The text description is mapped to a text feature vector by a text encoder, which is used for semantic and visual feature space alignment in the image generation base model. The attention layer parameters of the image generation base model are adjusted according to the weight allocation of the key information; The main body region and background region of the image generation basic model are determined based on the positional relationship of the main object information, environmental scene information, and action interaction information.

10. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the image generation method according to any one of claims 1 to 7.

Images