Image generation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an image generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a reference image and respective control information of each reference object in the reference image, the control information representing an adjustment condition of the corresponding reference object under at least one specified adjustment dimension; obtaining a corresponding low-rank adaptive weight set based on the control information associated with each reference object; each denoising network layer is associated with at least one adjustment dimension; adjusting network parameters of each denoising network layer in a pre-trained image generation model based on each obtained low-rank adaptive weight set, and obtaining a trained image generation model; and inputting the reference image and each piece of control information into a trained image generation model to obtain a condition image generated by combining the respective mask area of each reference object. Through layered and regional control, conflicts among lora are avoided, and the accuracy of the generated image is improved.

Description

Background Art

[0002] Low-Rank Adaptation (LoRa) is a technology developed by researchers to solve the problem of fine-tuning large language models. This technology can achieve fine control over the image generation model, making it easy to draw specific people, objects, special brushstrokes, special painting styles, etc.

[0003] The conditional image generation task in the related technology is generally not met by a single lora. For example, if you want to generate an image of an anime-style red panda wearing a red skirt, you need to combine three loras, namely: red panda character lora, clothes lora, and anime style lora.

[0004] When multiple LoRa images are needed to generate conditional images, the related technology simply merges multiple LoRa images. However, in this way, conflicts will occur between multiple LoRa images. For example, the LoRa image of the red panda character is more focused on the second dimension, while the LoRa image of clothing is more focused on the realistic style. The conflict between LoRa images will lead to unsatisfactory results. Therefore, how to accurately generate conditional images that meet expectations is an urgent problem to be solved. Summary of the invention

[0005] Embodiments of the present application provide an image generation method, device, electronic device, and storage medium to improve the accuracy of generating conditional images.

[0006] An image generation method provided in an embodiment of the present application includes:

[0007] Acquire a reference image and control information of each reference object in the reference image, wherein the control information indicates: an adjustment condition of the corresponding reference object in at least one specified adjustment dimension;

[0008] For each reference object, the following operations are performed respectively: based on control information associated with a reference object, a low-rank adaptive weight set is obtained; the low-rank adaptive weight set includes: low-rank adaptive weights of each denoising network layer included in the pre-trained image generation model, wherein each denoising network layer is associated with at least one adjustment dimension;

[0009] Based on the obtained low-rank adaptive weight sets, adjusting the network parameters of each denoising network layer in the pre-trained image generation model to obtain a trained image generation model;

[0010] The reference image and the obtained control information are input into the trained image generation model to obtain a conditional image generated by adjusting the reference image in combination with the respective mask areas of the reference objects; each mask area is determined based on the position information of the corresponding reference object in the reference image.

[0011] An image generating device provided in an embodiment of the present application includes:

[0012] A data acquisition unit, configured to acquire a reference image and control information of each reference object in the reference image, wherein the control information indicates an adjustment condition of the corresponding reference object under at least one specified adjustment dimension;

[0013] The weight prediction unit is used to perform the following operations for each reference object: based on the control information associated with a reference object, obtain a low-rank adaptive weight set; the low-rank adaptive weight set includes: the low-rank adaptive weights of each denoising network layer included in the pre-trained image generation model, wherein each denoising network layer is associated with at least one adjustment dimension;

[0014] A model adjustment unit, configured to adjust network parameters of each denoising network layer in the pre-trained image generation model based on each obtained low-rank adaptive weight set, so as to obtain a trained image generation model;

[0015] An image generation unit is used to input the reference image and the obtained control information into the trained image generation model to obtain a conditional image generated by adjusting the reference image in combination with the respective mask areas of the reference objects; each mask area is determined based on the position information of the corresponding reference object in the reference image.

[0016] Optionally, the model adjustment unit is specifically used to:

[0017] For each denoising network layer, perform the following operations:

[0018] Performing weighted summation on the low-rank adaptive weights of each reference object for a denoising network layer to obtain a comprehensive low-rank adaptive weight of the denoising network layer;

[0019] Based on the comprehensive low-rank adaptive weight of the denoising network layer, adjusting the weight matrix of the denoising network layer in the pre-trained image generation model to obtain a corresponding adjusted weight matrix;

[0020] The trained image generation model is determined according to the adjusted weight matrices of each denoising network layer in the pre-trained image generation model.

[0021] Optionally, the model adjustment unit is further used for:

[0022] Before adjusting the network parameters of each denoising network layer in the pre-trained image generation model based on the obtained low-rank adaptive weight sets to obtain the trained image generation model, obtaining initial weights set for each reference object, each initial weight is positively correlated with the similarity between the corresponding reference object and the expected adjustment result;

[0023] The model adjustment unit is specifically used for:

[0024] Based on the initial weights of the reference objects, the low-rank adaptive weights of the denoising network layer are weighted summed to obtain a comprehensive low-rank adaptive weight of the denoising network layer.

[0025] Optionally, the model adjustment unit is specifically used to:

[0026] For each denoising network layer, the following operations are respectively performed: adjusting a weight matrix corresponding to the denoising network layer in the pre-trained image generation model according to the low-rank adaptive weights of each reference object for the denoising network layer, to obtain a corresponding adjusted weight matrix;

[0027] The trained image generation model is determined according to the adjusted weight matrices of each denoising network layer in the pre-trained image generation model.

[0028] Optionally, the weight prediction unit is specifically used for:

[0029] Inputting the control information associated with the reference object into a trained weight prediction model to obtain a low-rank adaptive weight set output by the trained weight prediction model;

[0030] Wherein, the trained weight prediction model is obtained through reinforcement learning training.

[0031] Optionally, the device is also used for:

[0032] The model training unit is used to train the trained weight prediction model in the following manner:

[0033] Obtain a training sample set, wherein each training sample in the training sample set includes: a candidate state, and a true value of a long-term reward corresponding to each candidate adjustment behavior under the candidate state; the candidate state represents: a candidate low-rank adaptive weight for each denoising network layer in the pre-trained image generation model; each candidate adjustment behavior represents: an adjustment amount when adjusting the candidate low-rank adaptive weight;

[0034] Based on the training sample set, a weight prediction model to be trained is subjected to cyclic iterative training to obtain the trained weight prediction model, wherein each iterative training performs the following process:

[0035] Selecting training samples from the training sample set, and inputting candidate states in the training samples into the weight prediction model to be trained, to obtain the long-term return estimation value corresponding to each adjustment behavior under the candidate state;

[0036] Based on the difference between each obtained long-term reward estimate and the corresponding long-term reward true value, the parameters of the weight prediction model to be trained are adjusted.

[0037] Optionally, the model training unit is specifically used for:

[0038] The following learning process is iteratively performed based on the sample image set until the amount of data in the memory pool reaches a preset observation value, and then a portion of data is randomly selected from the memory pool to construct the training sample set; wherein each learning process performs the following operations:

[0039] Selecting a sample image from the sample image set, and setting an initial state according to a sample adjustment dimension of the sample image, wherein the initial state represents: initial low-rank adaptive weights of each denoising network layer in the pre-trained image generation model;

[0040] Selecting an initial adjustment behavior according to the initial state, the initial adjustment behavior representing: an adjustment amount when adjusting the initial low-rank adaptive weight;

[0041] After performing the initial adjustment behavior on the pre-trained image generation model, a reward value and a candidate state after performing the initial adjustment behavior are obtained; the reward value represents: a quality change of an image generated by the image generation model before and after the adjustment;

[0042] Save the initial state, initial adjustment behavior, reward value and candidate state to the memory pool.

[0043] Optionally, the model training unit is further used for:

[0044] In each learning process, if the reward value meets the preset threshold condition, the learning process ends;

[0045] If the reward value does not meet the preset threshold condition, the candidate state is used as the initial state, and the process returns to the step of selecting an initial adjustment behavior according to the initial state.

[0046] Optionally, the denoising network layer is a cross-attention layer included in the denoising network in the image generation model; the cross-attention layer is used to model the connection between control information and the image.

[0047] Optionally, the image generating unit is specifically used for:

[0048] Inputting the reference image into the trained image generation model, encoding the reference image based on the encoding layer in the trained image generation model to obtain an encoded image;

[0049] Randomly adding noise to the coded image to obtain a noisy image;

[0050] Encoding the control information through a conditional network in the trained image generation model to obtain conditional coding features;

[0051] Inputting the conditional coding features and the noise image into the denoising network in the trained image generation model, performing iterative denoising processing in combination with the mask region, and obtaining a denoised image;

[0052] Based on the decoding layer in the trained image generation model, the denoised image is decoded to obtain the conditional image.

[0053] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned image generation methods.

[0054] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of any one of the above-mentioned image generation methods.

[0055] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned image generation methods.

[0056] The beneficial effects of this application are as follows:

[0057] The embodiments of the present application provide an image generation method, device, electronic device and storage medium. The present application takes into account the connection between each denoising network layer contained in the pre-trained image generation model and the adjustment dimension of the image, and proposes the idea of ​​controlling the control weight of each denoising network layer in the pre-trained image generation model by LoRa, that is, in the present application, multiple LoRas are not simply fused, but different denoising network layers are controlled by the hierarchical setting of LoRa weights (i.e., low-rank adaptive weights) to achieve control of different dimensions of the image. Specifically, after obtaining the control information of the reference image and each reference object in the reference image, the present application considers that each of these control information corresponds to at least one specified adjustment dimension, and therefore, based on these control information, a low-rank adaptive weight set for the pre-trained image generation model can be determined, and the weight set includes the low-rank adaptive weights of each denoising network layer contained in the pre-trained image generation model, so as to hierarchically lock the LoRa weights and avoid conflicts between LoRas. The trained image generation model obtained on this basis can achieve control of different dimensions of the image. In addition, when generating conditional images based on the generated trained image generation model, the present application further introduces the feature of the mask area related to the reference object. The scope of action of LoRa is determined by the mask area. On the basis of LoRa stratification, the effect of LoRa regional control is further realized, thereby avoiding conflicts between LoRas and improving the accuracy of the generated conditional images.

[0058] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0060] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;

[0061] Figure 2 is a flowchart of an implementation method of an image generation method in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of a setting interface of an image production platform in an embodiment of the present application;

[0063] Figure 4A schematic diagram of a setting interface of another image production platform in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of a denoising network Unet in an embodiment of the present application;

[0065] Fig. 6A A schematic diagram of a hierarchical control result in an embodiment of the present application;

[0066] Figure 6B A schematic diagram of another hierarchical control result in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of a sub-region control flow chart in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of a process of automatic search for lora hierarchical control weights in an embodiment of the present application;

[0069] Fig. 9 A training flow chart of a weight prediction model in an embodiment of the present application;

[0070] Fig.10 A flowchart of a reinforcement learning process in an embodiment of the present application;

[0071] Fig.11 A conditional image generation flow chart in an embodiment of the present application;

[0072] Fig.12 A schematic diagram of the overall framework of an image generation model in an embodiment of the present application;

[0073] Fig.13A This is a schematic diagram of automatic merging of multiple LoRas in an embodiment of the present application;

[0074] Fig. 13B This is another schematic diagram of multi-LoRa automatic merging in an embodiment of the present application;

[0075] Fig.14 This is a schematic diagram of multiple lora superposition results in an embodiment of the present application;

[0076] Fig.15 A schematic diagram of the structure of an image generating device in an embodiment of the present application;

[0077] Fig.16 A schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;

[0078] Fig.17 The present invention is a schematic diagram of the hardware structure of another electronic device to which the embodiments of the present application are applied. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0080] The following is an introduction to some concepts involved in the embodiments of the present application.

[0081] 1. Pre-training: refers to obtaining a pre-trained model that is not related to a specific task from large-scale data through self-supervised learning. On this basis, it can be further trained or used for other purposes. The training and use of pre-trained models correspond to two stages: pre-training and fine-tuning. In the pre-training stage, the model is generally trained in an unsupervised or weakly supervised manner on a very large corpus, and it is expected that the model can obtain language-related knowledge, such as syntax, grammatical knowledge, etc., to reflect the semantic representation of a word in a specific context. The fine-tuning stage uses the pre-trained model to customize training for certain tasks and correct the network for specific tasks. The training method of the pre-trained model can use self-supervised learning technology (such as autoregressive language models and autoencoding technology).

[0082] The image generation model in the embodiment of the present application is a pre-trained model. After setting the lora weights of each denoising network layer in the image generation model by the method in the embodiment of the present application, a trained image generation model can be obtained for conditional image generation tasks.

[0083] 2. Low-rank adaptive weights: Lora weights. Lora is the low-rank adaptation of large language models. It freezes the weights of the pre-trained model and injects the trainable rank decomposition matrix into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. In an embodiment of the present application, Lora mainly injects trainable network parameters into the denoising network layer in the image generation model. The denoising network layer is used to link image representations with control information that describes them, while the Lora weights affect the network parameters corresponding to the denoising network layer, such as the weight matrix part of the denoising network layer.

[0084] 3. Reinforcement Learning (Q-Learning): The subject is an agent, which can obtain the state of the environment, take actions in the environment to affect the environment, and finally obtain rewards from the environment. Reinforcement learning is a process of obtaining the maximum reward by finding the best sequence of actions. Specifically, robots that learn and make decisions are usually called agents, and all things that interact with them are called environments. The agent selects actions, and the environment responds to these actions and presents a new state to the agent. At the same time, the environment will also generate a benefit (i.e., reward), which is the goal that the agent wants to maximize in the process of selecting actions. This series of decision-making processes can be modeled as a Markov decision process (MDP), which is a mathematical idealization of reinforcement learning problems and a theoretical framework for achieving goals through interactive learning.

[0085] 4. Control information: refers to the specific generation conditions in the conditional image generation task. For example, when you want to generate an image of a red panda in anime style wearing a red skirt, the control information is "a red panda in anime style wearing a red skirt". Specifically, the control information in the embodiments of the present application can be divided according to the reference object. The reference object refers to the part of the reference image that needs to be adjusted. Therefore, each reference object corresponds to a control information, which means that different adjustment parts are described by different control information.

[0086] 5. Expected adjustment result: refers to the conditional image that is expected to be generated. Specifically, when there are multiple reference objects in the reference image, the expected adjustment result also includes the parts corresponding to each reference object. For example, when there are 4 girls in the reference image, representing 4 reference objects, the corresponding expected adjustment result also includes the result parts corresponding to each of the 4 girls. For another example, when the face and clothes of the person in the reference image are adjusted, and the face and clothes are each used as a reference object, the corresponding expected adjustment result also includes the result parts corresponding to each of the two parts.

[0087] 6. Diffusion model: It is a generative model. The intuition behind the diffusion model comes from physics. In physics, gas molecules diffuse from high-concentration areas to low-concentration areas, which is similar to the loss of information due to noise interference. So by introducing noise, and then trying to generate an image by denoising. Through multiple iterations over a period of time, the model learns to generate new images each time given some noise input. The image generation model in the embodiment of the present application can use the diffusion model as the basis.

[0088] The word “exemplary” is used hereinafter to mean “serving as an example, example, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0089] The terms "first" and "second" in this document are used for descriptive purposes only and should not be understood as explicitly or implicitly indicating relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0090] The embodiments of the present application relate to artificial intelligence (AI) and machine learning technology, and are designed based on machine learning (ML) in artificial intelligence.

[0091] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0092] Artificial intelligence is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation interaction systems, mechatronics and other technologies; the software technologies of artificial intelligence generally include computer vision technology, natural language processing technology, and machine learning / deep learning and other major directions. With the development and progress of artificial intelligence, artificial intelligence is being studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, robots, smart medical care, etc. It is believed that with the further development of future technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0093] Machine learning is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0094] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Deep learning is the core of machine learning and a technology to achieve machine learning. Machine learning generally includes technologies such as deep learning, reinforcement learning, transfer learning, and inductive learning. Deep learning includes technologies such as mobile visual neural network Mobilenet, convolutional neural network (CNN), deep belief network, recursive neural network, autoencoder, generative adversarial network, etc. The training of the weight prediction model in the embodiment of the present application involves reinforcement learning technology.

[0095] Reinforcement learning, also known as reinforcement learning and evaluation learning, is an important machine learning method with many applications in the fields of intelligent control robots and analysis and prediction. In the embodiment of the present application, the weight prediction model is trained by reinforcement learning technology, and then, a better weight combination result can be automatically searched based on the trained weight prediction model to achieve lora weight hierarchical locking, so as to control different denoising network layers and realize the control of different dimensions of the image.

[0096] The following is a brief introduction to the design concept of the embodiment of the present application:

[0097] For large models, retraining all model parameters to fine-tune becomes infeasible. For example, GPT-3 (Generative Pre-trained Transformer 3) is a natural language processing model based on artificial intelligence technology. The model contains 175B parameters, and both fine-tuning training and model deployment are impossible. So Microsoft proposed lora, which freezes the pre-trained model weights and injects the trainable rank decomposition matrix into each layer of the Transformer (Transformer is the T in GPT) architecture. Since there is no need to recalculate the gradient of the model's weight parameters, the number of trainable parameters for downstream tasks is greatly reduced.

[0098] In the related art, conditional image generation tasks are generally not met by a single lora. Currently, multiple loras are simply fused and the weight of each lora is controlled to adjust the effects of different loras. In this case, conflicts between loras will lead to unsatisfactory generation results, and the weight of lora needs to be manually adjusted, which is not only time-consuming, but also cannot guarantee the global optimal result.

[0099] In view of this, the embodiments of the present application propose an image generation method, device, electronic device and storage medium. The present application takes into account the connection between each denoising network layer contained in the pre-trained image generation model and the adjustment dimension of the image, and proposes the idea of ​​controlling the control weight of LoRa on each denoising network layer in the pre-trained image generation model. That is, in this application, multiple LoRas are not simply fused, but different denoising network layers are controlled through the hierarchical setting of LoRa weights to achieve control of different dimensions of the image.

[0100] Specifically, after obtaining the control information of the reference image and each reference object in the reference image, the present application takes into account that each of the control information corresponds to at least one specified adjustment dimension. Therefore, based on the control information, a low-rank adaptive weight set for the pre-trained image generation model can be determined. The weight set includes the low-rank adaptive weights of each denoising network layer contained in the pre-trained image generation model, so as to hierarchically lock the LoRa weights and avoid conflicts between LoRas. The trained image generation model obtained on this basis can realize control of different dimensions of the image.

[0101] In addition, when generating conditional images based on the generated trained image generation model, the present application further introduces the feature of the mask area related to the reference object. The scope of action of LoRa is determined by the mask area. On the basis of LoRa stratification, the effect of LoRa regional control is further realized, thereby avoiding conflicts between LoRas and improving the accuracy of the generated conditional images.

[0102] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0103] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

[0104] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; the terminal device may be installed with a client related to image processing, which may be software (such as a browser, AI drawing software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for image processing, which is not specifically limited in the present application. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0105] It should be noted that the image generation method in each embodiment of the present application can be executed by an electronic device, and the electronic device can be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed by the terminal device 110 and the server 120 together. The following embodiments are described by taking the server 120 executing alone as an example.

[0106] In some application scenarios, drawing software may be installed on the terminal device 110, and the conditional image generator may select a reference image through the drawing software, and set the control information corresponding to each reference object in the reference image. Then, the terminal device 110 may send the reference image and the control information of each reference object in the reference image to the server 120, and then, the server 120 may use the image generation method of the embodiment of the present application to process and obtain the conditional image. Finally, the server 120 may feed back the conditional image to the terminal device 110, and the terminal device 110 presents it to the object.

[0107] It should be noted that the above is a simple example of the joint execution of the terminal device 110 and the server 120. The process is similar when the terminal device 110 or the server 120 executes it alone, which will not be described in detail here.

[0108] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network.

[0109] In an optional implementation, the communication network is a wired network or a wireless network.

[0110] It should be noted that Figure 1The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0111] In an embodiment of the present application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as in the image generation method disclosed in the embodiment of the present application, the image-related data involved can be saved on the blockchain, for example, reference images, control information, low-rank adaptive weights, conditional images, etc.

[0112] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.

[0113] The following describes the image generation method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0114] See also Figure 2 As shown, it is an implementation flow chart of an image generation method provided in an embodiment of the present application. Taking the server as the execution subject as an example, the specific implementation process of the method is as follows S21-S24:

[0115] S21: The server obtains a reference image and control information of each reference object in the reference image.

[0116] Specifically, the reference object refers to the part of the reference image that needs to be adjusted, such as a person, an animal, an object, a picture background, etc. Different reference objects can be different people, different animals, different objects, or different parts of the same person / animal / object, etc., which is not specifically limited herein.

[0117] The control information is the adjustment condition of the conditional image generation task. Each reference object may correspond to a control information, which indicates the adjustment condition of the corresponding reference object under at least one specified adjustment dimension. The "at least one" may be understood as one or more, and the "more" may be understood as two or more than two, such as two, three, etc.

[0118] In an embodiment of the present application, the specified adjustment dimension corresponding to the reference object is specific, that is, it specifically refers to the specific adjustment dimension for the reference object in this conditional image generation task, one or more of the adjustment dimensions supported by the category of the reference object, which is related to the expectations of the conditional image generator.

[0119] In an embodiment of the present application, the supported adjustment dimensions may be different for reference objects of different categories, that is, the adjustment dimensions in the embodiment of the present application are related to the object category, and the adjustable parts of reference objects of different categories are different, and the adjustment dimensions are obtained by dividing the adjustable parts of the object.

[0120] For example, when the reference object is a person or animal, it can be broken down into the following adjustment dimensions, including but not limited to the following:

[0121] 1) Appearance of the character / animal: including the character / animal's hairstyle, eyes, face shape, mouth, expression, etc.; 2) Clothing of the character / animal: including headwear, clothing, shoes, accessories, etc.; 3) Posture and action of the character / animal: the posture or action of the character / animal; 4) Picture style: the picture style of the entire Lora; 5) Picture background: the background of the entire Lora picture.

[0122] For another example, when the reference object is an object, it can be decomposed into the following adjustment dimensions, including but not limited to the following:

[0123] 1) Object appearance; 2) Object material; 3) Picture style: the picture style of the entire Lora; 4) Picture background: the background of the entire Lora picture.

[0124] It should be noted that the adjustment dimensions corresponding to the objects in the embodiments of the present application are related to the categories of the objects. Each category of objects may correspond to a set of adjustment dimensions, which can be obtained based on which parts of the objects can be adjusted and disassembled. This article will not go into details one by one.

[0125] Specifically, the reference image and control information are generally set by the conditional image generator according to its own needs. As listed above, when the object wants to obtain an image of an A-anime-style red panda wearing a red skirt, the reference image can be an image of a red panda not wearing a red skirt and not in A-anime style, or it can be other images containing an animal, person, virtual character, etc., which is not specifically limited in this article.

[0126] In the above case, there is only one reference object in the reference image, namely, the red panda. The control information corresponding to the reference object is "a cartoon-style red panda wearing a red skirt". The corresponding specified adjustment dimensions include clothing and picture style.

[0127] For example, if you want to generate a character with blue eyes and a red skirt based on reference object A, the blue eyes belong to the character's appearance dimension and the red skirt belongs to the character's clothing dimension. That is, there are two designated adjustment dimensions corresponding to reference object A, namely, character appearance and character clothing.

[0128] For another example, when you want to generate a character with both hands raised based on reference object B, "raising both hands" belongs to the character posture action dimension, that is, reference object B has only one designated adjustment dimension corresponding to it, namely, the character posture action. And so on.

[0129] In addition, it should be noted that the control information in the embodiments of the present application may be in the form of text, image, voice, etc., which is not specifically limited herein.

[0130] See also Figure 3 As shown, it is a schematic diagram of a setting interface of an image production platform in an embodiment of the present application. Figure 3 Part 31 illustrates the selection of a reference image and the setting of related control information.

[0131] Among them, through the selection of IP, an image as a reference image can be set, as shown in part 32, by default, all images in the image production platform gallery are selected, and the conditional image generator can set to cancel all selections, and then select one from the current gallery as a reference image, or the conditional image generator can also choose to browse other images except the image production platform gallery, which is not specifically limited in this article. Further, through the selection of facial features, expressions, styles, clothing, backgrounds, actions, etc., control information can be set.

[0132] Among them, "background" is generally a noun / adjective, etc., separated by commas. For example, "eg: blue sky, white clouds, beach". "Action" is generally a verb, separated by commas, such as "eg: look back, open arms".

[0133] In addition, the subject can also add other feature descriptions of the expression image, expression background, etc. in the "Description" module, generally in the form of phrases separated by commas, such as "eg: playing with mobile phone, smiling happily in the sun".

[0134] The object can also select the size and number of images to be generated. Figure 3 The number is "5", which means that 5 conditional images that meet the above conditions need to be generated.

[0135] In an embodiment of the present application, the conditional image generator can trigger the image generation process in the embodiment of the present application by clicking a "Generate" button, and the terminal device sends the reference image and various control information set by the conditional image generator to the server, and the server executes the image generation method in the embodiment of the present application to generate a conditional image that meets the above conditions.

[0136] Optionally, if the conditional image generator is not satisfied with the reference image or control information set above, it can click the "Reset" button to reselect the IP, facial features, expressions, style, etc. This article does not make specific restrictions.

[0137] See also Figure 4 As shown, it is a schematic diagram of another setting interface of the image production platform in the embodiment of the present application. Figure 4 The following is the expression AIGC page of the image production platform, which is the setting page of the reference image and control information related to this application. Common condition configurations include: IP, facial features, expression, style, clothing, background, action, quantity, size, description, etc.

[0138] Among them, AIGC (AI Generated Content) refers to the use of artificial intelligence technology to generate content. AIGC is a new production method that uses AI technology to automatically generate content. AI painting, AI writing, etc. are all branches of AIGC. The image generation method in the embodiment of this application mainly refers to AI painting.

[0139] Among them, "background" is usually a noun / adjective, etc., separated by commas, such as "eg: sea, gray sky". "Action" is usually a verb, separated by commas, such as "eg: selfie, dance, run".

[0140] same Figure 3 same, Figure 4 The "description" module in the expression can be supplemented with other feature descriptions of the expression image, expression background, etc., generally in the form of phrases separated by commas, such as "eg: blue clothes, wide eyes, drinking coffee in the sun".

[0141] Similarly, the conditional image generator can trigger the image generation process in the embodiment of the present application by clicking the "Generate" button, and can also reselect the IP, facial features, expressions, style, etc. by clicking the "Conditional Clear" button, which will not be repeated here.

[0142] in, Figure 4 The control information in the video is in the form of text or image. For example, expressions and clothing are in the form of images, while background, style, and action are in the form of text.

[0143] In addition, it should be noted that the above examples are based on the example that the reference image contains only one reference object. When there are multiple reference objects, the setting method is the same as the above. Figure 3 , Figure 4 Similarly. Specifically, when setting control information for each reference object, you can first select the reference object to be set this time, and on this basis, further set the control information such as expression, clothing, action, facial features, etc.; or, when setting relevant control information in a certain column, note the reference object's logo, etc. Any setting method is applicable to the embodiments of the present application, and this article does not make specific limitations.

[0144] It should be noted that the above-listed reference image and control information setting methods are just simple examples, and any setting method is applicable to the embodiments of the present application and is not specifically limited here.

[0145] S22: For each reference object, the server performs the following operations respectively: based on control information associated with a reference object, obtain a low-rank adaptive weight set.

[0146] The low-rank adaptive weight set includes: the low-rank adaptive weights of each denoising network layer included in the pre-trained image generation model.

[0147] In the embodiment of the present application, each denoising network layer is associated with at least one adjustment dimension. The present application formally considers the connection between each denoising network layer and the adjustment dimension of the image, and proposes the idea of ​​controlling the control weights of each denoising network layer in the pre-trained image generation model by LoRa. That is, the present application does not simply fuse multiple LoRas, but controls different denoising network layers through the hierarchical setting of LoRa weights to achieve control of different dimensions of the image.

[0148] In an embodiment of the present application, when the image generation model is used to generate a conditional image, it is mainly divided into two processes: adding noise and removing noise. First, it is necessary to add noise to the reference image, such as randomly adding Gaussian noise. This process can be a fixed Markov chain process, and the original data distribution is changed to a normal distribution by continuously adding Gaussian noise. Then, the Gaussian noise is converted into the content of the known data distribution through an iterative denoising process, such as using a neural network to restore the data from a normal distribution to the original data distribution, and the generated image has good diversity and realism. In the denoising process, the denoising network layer in the embodiment of the present application is mainly used to model the relationship between control information and the image. Therefore, through the denoising network layer, the image is denoised with each control information as a condition, and a conditional image corresponding to the control information can be generated. The conditional image is an image that meets expectations. Specifically, the correlation between the image content of the conditional image and the content of each control information is greater than a certain threshold.

[0149] Among them, the image generation model is a model that has been pre-trained, and the image generation model can be an existing image generation model. This application does not specifically limit the network structure of the image generation model and will not elaborate on it.

[0150] Optionally, the denoising network layer is a cross-attention layer included in the denoising network in the image generation model; in an embodiment of the present application, the cross-attention layer is used to model the connection between control information and the image, and Lora can pay attention to the trainable part in the cross-attention layer. Therefore, by controlling the network parameters of the cross-attention layer through Lora and processing the image based on each control information, an expected conditional image can be obtained.

[0151] See also Figure 5 As shown, it is a schematic diagram of a denoising network Unet in the embodiment of the present application. Among them, Unet can be understood as a part of the image generation model in the embodiment of the present application (same Fig.12 1222 in the image), which is mainly used to denoise the image. Each network layer contained in Unet can be named as a denoising network layer according to the function of the network. Figure 5 The structures of different layers of Unet are listed, and due to space limitations, Figure 5 Only some of the denoising network layers are illustrated.

[0152] from Figure 5 It can be seen that the lora workflow is divided into three parts: 51 input part (IN), 52 middle part (MID) and 53 output part (OUT). In addition, there is an additional 54 switch BASE (first layer), specifically a text encoder.

[0153] In the embodiment of the present application, a text encoder is used to encode the control information. In this case, the control information should be in text form. If the control information is in image form, the image can be converted into text by image classification, semantic recognition, etc. If the control information is in voice form, the voice can be converted into text by voice recognition, etc.

[0154] That is, for control information in other forms than text, the control information may be first converted into text form and then encoded by a text encoder. Alternatively, a matching BASE may be set according to the form of the control information. For example, in addition to being set as a text encoder, BASE may also be set as an image encoder or other forms, which are not specifically limited herein.

[0155] Except for the switch BASE, the IN input part, the MID middle part and the OUT output part can all be understood as the denoising network layer in the image generation model.

[0156] like Figure 5As shown, the input part 51 simply illustrates only 4 layers, namely residual module 511, attention module 512, residual module 513 and attention module 514; the input part 52 simply illustrates only 3 layers, namely residual module 521, attention module 522, residual module 523; the input part 53 simply illustrates only 4 layers, namely residual module 531, attention module 532, residual module 533 and attention module 534.

[0157] It should be noted that Figure 5 The Unet structure listed is just a simple example. The following example takes the above parts plus switches as an example, which have a total of 17 layers. Specifically, among these 17 layers, the switch is the first layer, the IN input part is the 2-7 layers, the MID middle part is the 8th layer, and the OUT output part is the 9-17 layers.

[0158] In the embodiment of the present application, the 17 layers can be classified into 7 subcategories, as shown in Table 1 below:

[0159] Table 1 Examples of several lora weight settings

[0160] INS 17 floors 2 to 4, the first floor is BASE 1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0 IND 17 floors 5 to 7, the first floor is BASE 1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0 INALL INS+IND, the entire IN part, 2-7 layers 1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0 MIDD IND+MID+OUTD part, 5th to 12th floors 1,0,0,0,1,1,1,1,1,1,1,1,0,0,0,0,0 OUTD 17 floors, 9 to 12, the first floor is BASE 1,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0 OUTS 17 floors, 13 to 17, the first floor is BASE 1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1 OUTALL OUTD+OUTS, the entire OUT part 1,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1

[0161] In summary, the IN input part is divided into two sub-parts: INS (17 layers 2 to 4) and IND (17 layers 5 to 7). The integration of these two sub-parts is INALL (17 layers 2 to 7), that is, the entire IN part. The OUT part is also divided into two sub-parts: OUTD (17 layers 9 to 12) and OUTS (17 layers 13 to 17). The integration of these two sub-parts is OUTALL (17 layers 9 to 17), that is, the entire OUT part. The MID part (17 layers 8), IND part (17 layers 5 to 7) and OUTD part (17 layers 9 to 12) can be recorded as the MIDD part (17 layers 5 to 12).

[0162] In an embodiment of the present application, each denoising network layer is associated with at least one adjustment dimension. Specifically, multiple denoising network layers may be associated with the same adjustment dimension, and one denoising network layer may be associated with one or more adjustment dimensions.

[0163] For example, the IN part focuses more on the control of image layout (such as object shape), and the INS part is relative to the IND part. For example, the INS part has a lower control over shape semantics than the IND part, while the IND part has a higher control over shape semantics than the INS part. The MIDD part is more biased towards the image of the object (such as appearance), and the OUT part focuses more on chroma. The OUTD part is also relative to the OUTS part. The closer to the middle layer, the higher the control over the image. For example, the OUTD part can control the image in addition to focusing on chroma relative to the OUTS part.

[0164] Taking the lora weight value of 0 to 1 as an example, since the relationship between control information and image needs to be modeled under the conditional image generation task, the weight of the BASE layer should be kept at 1. On this basis, if you want to adjust the shape of the reference object in the image, and the adjustment force is small, it can be considered that only the INS part is needed to play a role in the image denoising process, and the lora weights of each layer can be set to “1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0”; for example, if you want to adjust the shape of the reference object in the image, and the adjustment force is large, it can be considered that only the IND part is needed to play a role in the image denoising process, and the lora weights of each layer can be set to “1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0”; for example, if you want to adjust the shape of the reference object in the image, without considering other content, it can be considered that the INALL part is needed to play a role in the image denoising process, and the lora weights of each layer can be set to “1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0”.

[0165] See also Fig. 6A As shown, it is a schematic diagram of a hierarchical control result in an embodiment of the present application. It should be noted that, due to the limitation of drawing requirements, Figure 6 is a black and white line drawing, but in essence, these images shown in Figure 6 contain chromaticity information. In this application, different colors are represented by different filling patterns. In addition, for the convenience of description, the same filling pattern is used to represent the control results with similar colors, which are essentially slightly different. Fig. 6A For reference only. For example, if NONE is green, then IND, OUTD, OUTS, and OUTALL, which are close in color to NONE, are all greener than the results of other layers. For another example, if ALL is blue, then INALL and MIDD, which are close in color to ALL, are all bluer than the results of other layers.

[0166] like Fig. 6AAs shown in the figure, the corresponding image examples under several lora weight settings are listed respectively. Among them, NONE represents the original image, that is, the case where the lora weights of 17 layers are all 0; ALL represents the result when the lora weights of 17 layers are all 1; INS, IND, INALL, MIDD, OUTD, OUTS, and OUTALL correspond to the several weight settings listed in Table 1 above, and the corresponding results are as follows Fig. 6A shown.

[0167] See also Figure 6B As shown, it is a schematic diagram of another hierarchical control result in an embodiment of the present application. Figure 6B This is an example of adjusting the original image to a grayscale image, for reference only.

[0168] Depend on Fig. 6A and Figure 6B It can be seen that the initial layer of Lora affects the layout of the image, the subsequent layers have a greater impact on the color hue of the image, and the middle layers have an impact on multiple dimensions of the image. This makes it easier for Lora to generate things beyond its specialty.

[0169] In summary, lowering the weight of the initial layer can add some common attributes (such as shape) to the cartoon character; lowering the weight of the later layers can make the cartoon character appear more common colors; increasing the weight of the later layers can increase skin texture, and so on.

[0170] It should be noted that the image generation model listed above has 1 BASE layer and 16 denoising network layers, which is just a simple example. In fact, this article does not specifically limit the structure of the image generation model. There may be more denoising network layers (such as 21 layers), or fewer denoising network layers (such as 10 layers), which will not be elaborated here.

[0171] In the embodiment of the present application, different denoising network layers are controlled by the hierarchical setting of lora weights to achieve control of different dimensions of the image.

[0172] S23: The server adjusts the network parameters of each denoising network layer in the pre-trained image generation model based on the obtained low-rank adaptive weight sets to obtain a trained image generation model.

[0173] In an embodiment of the present application, some network parameters in the pre-trained image generation model are locked, for example, the weight parameters of other network layers except each denoising network layer remain unchanged by freezing, and LoRa affects only the denoising network layer, because the denoising network layer mainly models the relationship between the image and the text representation. The text representation in this application mainly refers to the text representation of the control information, that is, the denoising network layer is mainly used to model the relationship between the image and the control information, so as to combine the control information with the reference image to generate a target image that meets the expected requirements.

[0174] It should be noted that each reference object in the embodiment of the present application corresponds to a low-rank adaptive weight set (LoRa weight set, which contains the LoRa weights corresponding to each denoising network layer in the image generation model).

[0175] When there is only one reference object in the reference image, there is a corresponding lora weight set. For the same denoising network layer, the denoising network layer only corresponds to one lora weight. It is only necessary to adjust the weight matrix of a denoising network layer based on one lora weight.

[0176] When there are multiple reference objects in the reference image, there are multiple lora weight sets. For the same denoising network layer, the denoising network layer will correspond to multiple lora weights. The weight matrix of the same denoising network layer needs to be adjusted based on multiple lora weights. In this case, there are many specific adjustment methods, which are explained below with examples:

[0177] An optional implementation is that S23 can be implemented according to the following process, including the following sub-steps S231'-S233' ( Figure 2 not shown):

[0178] S231': For each denoising network layer, weighted sum is performed on the low-rank adaptive weights of each reference object for the denoising network layer to obtain a comprehensive low-rank adaptive weight of the denoising network layer.

[0179] S232': For each denoising network layer, based on the comprehensive low-rank adaptive weight of the denoising network layer, adjust the weight matrix of the denoising network layer in the pre-trained image generation model to obtain a corresponding adjusted weight matrix.

[0180] S233': Determine the trained image generation model according to the adjusted weight matrix of each denoising network layer in the pre-trained image generation model.

[0181] In the above implementation, multiple LoRa can be automatically merged by weighted summation. In this way, each denoising network layer corresponds to only one merged LoRa weight, that is, the comprehensive low-rank adaptive weight. Based on this, only one parameter adjustment is required for each denoising network layer, which is convenient and fast.

[0182] When implementing S231', it is necessary to perform weighted summation on multiple lora weights. Specifically, the weights may be set randomly, averaged, or set according to the requirements of the conditional image generator.

[0183] When the weights are set according to the requirements of the conditional image generator and step S231' is implemented based on the weights, an optional implementation is as follows:

[0184] The initial weights set for each reference object are obtained. Then, based on the initial weights of each reference object, the low-rank adaptive weights of a denoising network layer are weighted summed to obtain a comprehensive low-rank adaptive weight of a denoising network layer.

[0185] In an embodiment of the present application, each initial weight can be positively correlated with the similarity between the corresponding reference object and the expected adjustment result, that is, the initial weight can be set by the conditional image generator according to its own needs. For example, when you want to adjust the four characters in the reference image into two-dimensional characters, each character can be used as a reference object. The initial weight corresponding to each reference object represents the similarity between the character in the reference image and the two-dimensional character you want to obtain. The more similar they are, the larger the initial weight value.

[0186] Taking the above as an example, there are four reference objects in the reference image, and each reference object corresponds to a lora weight set. Assuming that there are 9 denoising network layers in the image generation model, each lora weight set contains 9 lora weight values, and each lora weight value corresponds to a denoising network layer.

[0187] Assume that the lora weight values ​​in the lora weight set corresponding to reference object 1 are {lora11, lora12, lora13, lora14, lora15, lora16, lora17, lora18, lora19}; the lora weight values ​​in the lora weight set corresponding to reference object 2 are {lora21, lora22, lora23, lora24, lora25, lora26, lora27, lora28, lora29}; the lora weight values ​​in the lora weight set corresponding to reference object 3 are {lora31, lora32, lora33, lora34, lora35, lora36, lora37, lora38, lora39}; the lora weight values ​​in the lora weight set corresponding to reference object 4 are {lora41, lora42, lora43, lora44, lora45, lora46, lora47, lora48, lora49}.

[0188] Assuming that the initial weights set for reference object 1 and reference object 2 are both 0.8, and the initial weights set for reference object 3 and reference object 4 are 1, taking the first denoising network layer as an example, the comprehensive low-rank adaptive weight of the denoising network layer is: lora1=0.8*(lora11+lora21)+1*(lora31+lora41); similarly, the comprehensive low-rank adaptive weight of the first denoising network layer is: lora2=0.8*(lora12+lora22)+1*(lora32+lora42),…, and so on.

[0189] Then, when adjusting the network parameters of the first denoising network layer, it can be based on lora1, and when adjusting the network parameters of the second denoising network layer, it can be based on lora2, and so on.

[0190] Specifically, based on a lora weight, when adjusting the network parameters of a denoising network layer, specifically referring to the adjustment of the weight matrix of the denoising network layer, the lora weight can be multiplied by the parameters in the original weight matrix Wa of the denoising network layer to obtain the adjusted weight matrix Wb. Taking the first denoising network layer as an example, Wb=lora1*Wa.

[0191] Alternatively, the sum of the original weight matrix Wa of the denoising network layer and the rank decomposition matrix Wlora determined based on the lora weight can be used as the adjusted weight matrix Wb. Taking the first denoising network layer as an example, Wb=Wa+Wlora.

[0192] It should be noted that the two methods listed above for adjusting the denoising network layer weight matrix based on low-rank adaptive weights are just simple examples. Any adjustment method is applicable to the embodiments of the present application and is not specifically limited here.

[0193] In the above implementation, the initial weights are set according to the needs of the conditional image generator, and lora merging is performed based on this. After the trained image generation model is obtained based on this, the conditional image obtained by the image generation model can better meet the needs of the conditional image generator, thereby improving the accuracy of the generated conditional image.

[0194] Another optional implementation is that S23 can be implemented according to the following process, including the following sub-steps S231"-S233" ( Figure 2 not shown):

[0195] S231”: For each denoising network layer, respectively adjust the weight matrix corresponding to the denoising network layer in the pre-trained image generation model according to the low-rank adaptive weights of each reference object for the denoising network layer, and obtain a corresponding adjusted weight matrix.

[0196] In the embodiment of the present application, when there are multiple reference objects, the weight matrix of the same denoising network layer needs to be adjusted three times. Still taking the above example, there are four reference objects in the reference image and nine denoising network layers in the image generation model.

[0197] Taking the above-mentioned adjustment method of multiplying the lora weight by the parameters in the original weight matrix of the denoising network layer as an example, the specific adjustment process is as follows: the first adjustment can obtain Wb1=lora11*Wa, the second adjustment can obtain: Wb2=Wb1+lora21*Wa, the third adjustment can obtain: Wb3=Wb2+lora31*Wa, and the fourth adjustment can obtain: Wb4=Wb3+lora41*Wa.

[0198] That is, the final adjusted weight matrix Wb is Wb4, Wb = (lora11 + lora21 + lora31 + lora41) * Wa.

[0199] In addition, in each adjustment, weighting may not be performed directly, but weighting may be performed in combination with the initial weights, so that the final Wb=(0.8*lora11+0.8*lora21+lora31+lora41)*Wa.

[0200] It should be noted that the order of the above four adjustment processes is not specifically limited, and the results are the same.

[0201] S232”: Determine the trained image generation model according to the adjusted weight matrices of each denoising network layer in the pre-trained image generation model.

[0202] This step means that after adjusting the weight matrix of each denoising network layer in the pre-trained image generation model in the above manner, the trained image generation model can be obtained.

[0203] In the above implementation, although the denoising network layer needs to be adjusted multiple times, this method still realizes the hierarchical setting of Lora weights. The hierarchical control method can effectively remove the overfitting part, avoid Lora conflicts, and solve the overfitting problem of Lora training.

[0204] S24: The server inputs the reference image and each obtained control information into a trained image generation model to obtain a conditional image generated by adjusting the reference image in combination with the respective mask regions of each reference object.

[0205] Each mask region is determined according to the position information of the corresponding reference object in the reference image. Specifically, the mask region can be determined according to the actual shape of the reference object, or a fixed shape such as a rectangular mask, a circular mask, etc., which is not specifically limited in this article.

[0206] Taking a rectangular mask as an example below, when using binary to mask an image, the pixel value of the rectangular area containing the reference object can be set to 0 (white) and the pixel value of other areas can be set to 1 (black) according to the position information of the reference object in the reference image.

[0207] like Figure 7 As shown, it is a schematic diagram of a sub-region control flow chart in an embodiment of the present application.

[0208] Figure 7 There are 4 people in the reference image shown, namely 4 reference objects. By extracting the person position information from the reference image, a mask area corresponding to each reference object can be generated. Figure 7 Part 72 of . On this basis, each reference object has its own corresponding lora weight set, which can be used to specify the lora corresponding to each mask through part 73 and connect them with AND.

[0209] As described below, this is an example of part 73 in the embodiment of the present application, and there are a total of 5 lines of code:

[0210] First line:

[0211] masterpiece,bestquality,((4girls)),multiplegirls,light smile,sea,beach,sunset

[0212] This line of code describes the reference image, specifically: Masterpiece, Best Quality, ((4 Girls)), Multiple Girls, Smile, Sea, Beach, Sunset.

[0213] Second behavior:

[0214] AND masterplece,best quality,4gjris,<lora:asoulAvapicolLORA_v10:0.8> ,

[0215] default,whitedress,wing hair ornament,twintalls

[0216] This line of code describes the lora weight set corresponding to the mask area of ​​the first reference object. The lines of code are connected by AND, specifically: and masterpiece, best quality, 4 girls,<lora:asoulAvapicolLORA_v10:0.8> , default, white dress, wing hair accessories, twin ponytails.

[0217] Among them, the content in <> is the name and initial weight of the lora weight set corresponding to the mask area of ​​the first reference object (that is, the initial weight of the reference object, which is used to calculate the comprehensive low-rank adaptive weight). "lora:asoulAvapicolLORA_v10" is the name, and 0.8 is the initial weight.

[0218] The third behavior:

[0219] AND masterplece,best quality,4glrls,<lora:asouleileenv20_v10:0.8> ,default dress,blue eyes,white hair,black dress

[0220] This line of code describes the lora weight set corresponding to the mask area of ​​the second reference object. The lines of code are connected by AND, specifically: and masterpiece, best quality, 4 girls,<lora:asouleileenv20_v10:0.8> , default dress, blue eyes, white hair, black dress.

[0221] The name of the LoRa weight set corresponding to the masked area of ​​this reference object is "LoRa:asouleileenv20_v10", and the initial weight is 0.8.

[0222] The fourth behavior is:

[0223] This line of code describes the lora corresponding to the mask area of ​​the third reference object. Each line of code is connected by AND, specifically:

[0224] AND masterpiece,best quality,4glrls,<lora:asoulbellav_v10:1> ,default,purple eyes,purple hair,ribbon

[0225] This line of code describes the lora corresponding to the mask area of ​​the third reference object. Each line of code is connected by AND, specifically: and masterpiece, best quality, 4 girls,<lora:asoulbellav_v10:1> , default, purple eyes, purple hair, ribbon.

[0226] The name of the Lora weight set corresponding to the mask area of ​​this reference object is "Lora:asoulbellav_v10" and the initial weight is 1.

[0227] The fifth line is:

[0228] AND masterplece,best quality,4glris,<lora:asoulDianapicoLORA_v10:1> ,blue eyes,brown hair,pink hair bowjacket,neckerchief

[0229] This line of code describes the lora corresponding to the mask area of ​​the fourth reference object. Each line of code is connected by AND, specifically: and masterpiece, best quality, 4 girls,<lora:asoulDianapicoLORA_v10:1> , purple eyes, brown hair, pink bow, jacket, scarf.

[0230] The name of the lora weight set corresponding to the mask area of ​​the reference object is "lora:asoulDianapicoLORA_v1", and the initial weight is 1.

[0231] In the above implementation, the influence range of LoRa can be controlled by masking the area to avoid conflicts between LoRa. In addition to the different areas corresponding to different characters listed above, different parts of the same object can also be set to correspond to different LoRa, such as character LoRa acting on the face, clothing LoRa acting on the body, etc., which is not specifically limited in this article.

[0232] Specifically, the S22 part in the image generation method of the present application represents the automatic search for the optimal weight of LoRa. In the embodiment of the present application, it supports the automatic search for the optimal weight of LoRa through positive and negative feedback. Among them, positive feedback refers to supporting the specified LoRa control dimension, and negative feedback refers to supporting manual adjustment.

[0233] See also Figure 8 As shown, it is a schematic diagram of the process of automatic search of lora hierarchical control weights in an embodiment of the present application.

[0234] like Figure 8 As listed, part 81 is the input image, and part 82 is the control result of the initial LoRa model (which can be understood as an image generation model generated based on a set of initial LoRa weight sets). On this basis, the conditional image generator can specify the LoRa control dimension. For example, if part 83 controls the color (positive feedback), the color-related weight values ​​in the LoRa model can be adjusted to ensure that the generated conditional image 84 is different in color from the input reference image and the same (or similar) as the color of the control result of the LoRa model. For example, the conditional image generator can also set negative feedback information according to demand. For example, if part 85 does not want to change the appearance, the weight values ​​related to the appearance in the LoRa model can be maintained to ensure that the generated conditional image 86 looks the same as the control result of the initial LoRa model.

[0235] Based on the above ideas, the embodiment of the present application proposes a method for automatically searching for the optimal lora weight based on reinforcement learning. Optionally, the acquisition of the lora weight set in the embodiment of the present application can be achieved through a reinforcement learning model. For each reference object, an optional implementation of step S22 is as follows:

[0236] Control information associated with a reference object is input into a trained weight prediction model to obtain a low-rank adaptive weight set output by the trained weight prediction model.

[0237] Alternatively, to further improve accuracy, a reference image and control information associated with a reference object may be input into a trained weight prediction model to obtain a low-rank adaptive weight set output by the trained weight prediction model.

[0238] Among them, the trained weight prediction model is obtained through reinforcement learning training. That is, the weight prediction model in the embodiment of the present application is a reinforcement learning model, which can be used to predict the lora weights of each denoising network layer according to the control information.

[0239] The control information in this application represents the adjustment conditions under at least one adjustment dimension. For the reference image, a specified adjustment dimension is set for each reference object. Therefore, based on the weight prediction model in this application, when the control information associated with the reference object is input, a low-rank adaptive weight set can be determined based on the specified adjustment dimension in the control information. The individual lora weight values ​​in the low-rank adaptive weight set correspond one-to-one to the denoising network layers contained in the image generation model.

[0240] In an embodiment of the present application, the larger the LoRa weight value of any denoising network layer, the stronger the role of the denoising network layer in the image generation process. Therefore, the size of the LoRa weight value in the low-rank adaptive weight set obtained in combination with the control information can be associated with the specified adjustment dimension corresponding to the reference object, ensuring that the LoRa weight corresponding to the denoising network layer associated with the specified adjustment dimension corresponding to the reference object is larger and plays a stronger role in the image generation process. Conversely, the LoRa weight corresponding to the denoising network layer that is irrelevant to the specified adjustment dimension corresponding to the reference object (or has a very small correlation) is smaller and plays a weaker role in the image generation process.

[0241] In the above implementation, the trained weight prediction model can be used to automatically predict the lora weight set associated with each reference object. This process does not require manual setting of the lora weights or multiple adjustments, and is more efficient and convenient.

[0242] Specifically, the basic idea of ​​learning the weight prediction model through reinforcement learning is Figure 8 Same as shown.

[0243] See also Fig. 9 As shown, it is a training flow chart of a weight prediction model in an embodiment of the present application, including the following steps S91-S92:

[0244] S91: Obtain a training sample set, wherein each training sample in the training sample set includes at least: a candidate state, and a true value of the long-term return corresponding to each candidate adjustment behavior under the candidate state.

[0245] Specifically, the candidate state represents: the candidate low-rank adaptive weights of each denoising network layer in the pre-trained image generation model; each candidate adjustment behavior represents: an adjustment amount when adjusting the candidate low-rank adaptive weights.

[0246] In the embodiment of the present application, the real value of the long-term reward is calculated based on the reward value (referring to the immediate reward) in the candidate state, or is pre-set based on experience. Specifically, the real value of the long-term reward represents the expected long-term reward that can be obtained by taking the candidate adjustment behavior (also called action) in the candidate state, which is equivalent to the expected value corresponding to each state action, which can be recorded as target_q and used for parameter adjustment of the subsequent weight prediction model.

[0247] For example, the candidate state in each training sample is recorded as s', each adjustment behavior in the candidate state can be recorded as action, and all adjustment behaviors can be recorded as all actions.

[0248] Each candidate state corresponds to an initial state (denoted as s), an initial adjustment behavior and a corresponding reward value (denoted as r).

[0249] Specifically, the candidate state is obtained by performing initial adjustment behavior on the initial state (which will be described in detail below), and the reward value represents the quality change between the first expected image generated by the image generation model determined based on the initial state and the second expected image generated by the image generation model determined based on the candidate state.

[0250] In the embodiment of the present application, the quality change can be specifically understood as a change in similarity between the second expected image and the expected adjustment result relative to the first expected image.

[0251] If the similarity between the first expected image and the expected adjustment result is b1, and the similarity between the second expected image and the expected adjustment result is b2, then r=b2-b1, which is the reward value corresponding to the candidate state s`.

[0252] It should be noted that the above-mentioned method of obtaining reward value is just a simple example. There may be other methods besides these, which are not specifically limited in this article.

[0253] When determining the true value of the long-term return based on the reward value, an optional calculation method is as follows:

[0254] Q(s,a)=r+Gamma*Max[Q(s`,all actions)](Formula 1)

[0255] Among them, Q(s,a) is a target_q under the candidate state s`, which can also be called the expected cumulative reward true value, r is the reward value corresponding to the candidate state s`, Gamma is a learning parameter, which can also be recorded as γ, and its value range is: 0≤γ<1. Specifically, γ represents the attenuation value of future r. The closer γ is to 1, the more sensitive the weight prediction model is to future r.

[0256] Max[Q(s`,all actions)] represents the maximum value of the q value determined under each candidate adjustment behavior. q represents the estimated long-term return, which is the result obtained based on the weighted prediction model.

[0257] It should be noted that when calculating the true value of the long-term return using the above method, under the same candidate state, the true value of the long-term return corresponding to each candidate adjustment behavior can be the same, that is, Q(s,a).

[0258] That is, under the same candidate state, a long-term return estimate can be obtained based on each candidate adjustment behavior. Therefore, there may be multiple long-term return estimates corresponding to the same long-term return true value.

[0259] Based on the above calculation method, we can further obtain the q value and the corresponding target_q value to train the weight prediction model (a neural network), as follows:

[0260] S92: Based on the training sample set, perform cyclic iterative training on the weight prediction model to be trained to obtain a trained weight prediction model.

[0261] Each iteration of training performs the following process:

[0262] S921: Select training samples from the training sample set, and input the candidate states in the training samples into the weight prediction model to be trained to obtain the long-term return estimation value corresponding to each adjustment behavior under the candidate state.

[0263] In the embodiment of the present application, the true value of the long-term reward can be recorded as target_q, and the estimated value of the long-term reward can be recorded as q. By taking the s' of all training samples as the input value of the weight prediction model to be trained and performing batch processing, the q value of each candidate adjustment behavior under the s' state can be obtained.

[0264] S922: Based on the differences between the obtained long-term reward estimates and the corresponding long-term reward true values, the parameters of the weight prediction model to be trained are adjusted.

[0265] Specifically, according to the difference between the q value obtained above and the target_q value, a loss function such as a cross entropy loss function, a mean square error loss function, etc. can be constructed, and then based on the constructed loss function, all parameters of the weight prediction model are updated through gradient descent and back propagation of the neural network, such as convolution kernel size, weight, bias parameters, etc., and activation function parameters can also be included. The specific situation depends on the actual situation and is not specifically limited in this article.

[0266] In an embodiment of the present application, combined with the long-term rewards in reinforcement learning, the parameters of the weight prediction model are adjusted through the difference between the true value of the long-term reward and the estimated value of the long-term reward. The training process does not require labeled data. It only needs to give an environment, a reward formula, and an automatically learning model to continuously improve itself. Compared with labeled training, the cost is much lower, and a lot of time for obtaining labeled samples is saved. In addition, the model has the ability of self-learning, and the training efficiency is higher.

[0267] An optional implementation is to obtain the training sample set in the following manner:

[0268] The following learning process is iteratively performed based on the sample image set until the amount of data in the memory pool reaches the preset observation value, and then some data is randomly selected from the memory pool to construct a training sample set; the specific learning process is as follows Fig.10 As shown, the following steps S101 to S108 are included:

[0269] S101: Select a sample image from a sample image set, and set an initial state according to a sample adjustment dimension of the sample image.

[0270] In the embodiment of the present application, the initial state represents: the initial low-rank adaptive weights of each denoising network layer in the pre-trained image generation model.

[0271] The sample adjustment dimension is for the sample image, which has the same meaning as the specified adjustment dimension in this article, and is specific, that is, the specific adjustment dimension for the sample object in the sample image. Specifically, the specific division of the adjustment dimension can be referred to in the above embodiment, and the repeated parts will not be repeated.

[0272] For example, an image generation model has 9 denoising network layers. For example, the sample adjustment dimensions of the selected sample image are the appearance and posture of the person. Assuming that, in the pre-trained image generation model, the denoising network layers related to these two adjustment dimensions are layers 2 to 5, when setting the initial state according to the sample adjustment dimension, only the initial low-rank adaptive weights of layers 2 to 5 are set to 1, and the initial low-rank adaptive weights of other denoising network layers are set to 0 (the default value of the BASE layer is 1).

[0273] For another example, the sample adjustment dimensions of the selected sample image are the appearance and clothing of the person. Assuming that, in the pre-trained image generation model, the denoising network layers related to these two adjustment dimensions are layers 3 to 6, then when setting the initial state according to the sample adjustment dimension, only the initial low-rank adaptive weights of layers 3 to 6 are set to 1, and the initial low-rank adaptive weights of other denoising network layers are set to 0 (the default value of the BASE layer is 1).

[0274] It should be noted that the above-mentioned process of setting the initial state according to the sample adjustment dimension of the sample image is just a simple example. The basic idea is: except for the BASE layer, the initial low-rank adaptive weights of the denoising network layer associated with the sample adjustment dimension are set relatively large, while the initial low-rank adaptive weights of the denoising network layer unrelated to the sample adjustment dimension are set relatively small.

[0275] For example, the initial low-rank adaptive weights of the denoising network layers associated with the sample adjustment dimension are all set to 1, and the initial low-rank adaptive weights of other denoising network layers are 0.

[0276] Alternatively, the initial low-rank adaptive weights of the denoising network layers associated with the sample adjustment dimension are all set to 0.5, the initial low-rank adaptive weights of other denoising network layers are 0, and so on.

[0277] Alternatively, the initial low-rank adaptive weights of the three denoising network layers associated with the sample adjustment dimension are set to 1, 0.9, and 0.8 respectively, and the initial low-rank adaptive weights of other denoising network layers are any values ​​in the range of 0 to 0.1, and so on.

[0278] It should be noted that the above-mentioned methods for setting the initial state are just simple examples and are not specifically limited in this article.

[0279] S102: Selecting an initial adjustment behavior according to the initial state.

[0280] The initial adjustment behavior represents: the adjustment amount when adjusting the initial low-rank adaptive weights.

[0281] It should be noted that in the embodiment of the present application, the value range of the low-rank adaptive weight is 0 to 1, that is, the minimum value is 0 and the maximum value is 1. When adjusting the low-rank adaptive weight, the adjustment amount can be flexibly set according to the actual situation and needs, such as +0.1 (maximum value is 1), -0.1 (minimum value is 0) for each adjustment; for example, +0.05 (maximum value is 1), -0.05 (minimum value is 0) for each adjustment, etc., which is not specifically limited in this article.

[0282] When selecting the initial adjustment behavior according to the initial state, if the initial low-rank adaptive weight of the second denoising network layer associated with the sample adjustment dimension is 1 and is currently the maximum value, then when setting the initial adjustment behavior, the initial adjustment behavior should be set to decrease by a certain value, such as -0.1.

[0283] For another example, when the initial low-rank adaptive weight of the second denoising network layer associated with the sample adjustment dimension is 0.5, it is not the maximum value at present. When setting the initial adjustment behavior, the initial adjustment behavior can be set to decrease by a certain value, such as -0.1, or to increase by a certain value, such as +0.1, etc. Alternatively, it can also be set by the -greedy method, etc., which is not specifically limited in this article.

[0284] S103: After performing an initial adjustment behavior on the pre-trained image generation model, a reward value and a candidate state after performing the initial adjustment behavior are obtained.

[0285] The reward value represents the change in quality of the image generated by the image generation model before and after adjustment.

[0286] The candidate state means the candidate low-rank adaptive weight obtained after the initial low-rank adaptive weight is adjusted according to the above initial adjustment behavior.

[0287] Specifically, when the initial low-rank adaptive weights of each denoising network layer in the initial state are adjusted based on the initial adjustment behavior, all denoising network layers can be adjusted, and different denoising network layers can be set with different initial adjustment behaviors (all layers are different or some layers are different); or, only the denoising network layers associated with the sample adjustment dimension this time can be adjusted. For these associated value-taking network layers, different denoising network layers can be set with different initial adjustment behaviors (all layers are different or some layers are different), or they can all be set to the same initial adjustment behavior, etc. This article does not make specific restrictions, and it can be set according to experience or randomly.

[0288] In an embodiment of the present application, performing the above-mentioned initial adjustment behavior on the pre-trained image generation model specifically refers to: adjusting the initial low-rank adaptive weights of each denoising network layer in the pre-trained image generation model based on the above-mentioned initial adjustment behavior, and on this basis, adjusting the weight matrix of the corresponding denoising network layer based on the adjusted low-rank adaptive weights to obtain the adjusted image generation model.

[0289] Still taking the above-mentioned image generation model with 9 denoising network layers as an example, assuming that before adjustment, the lora weights of each denoising network layer in the image generation model are 0, 1, 1, 1, 1, 0, 0, 0, 0. At this time, the first expected image can be generated according to the control information of the sample adjustment dimension corresponding to the image generation model before adjustment and the sample image.

[0290] After adjustment, the lora weights of each denoising network layer in the image generation model are 0, 0.9, 0.9, 0.9, 0.9, 0, 0, 0, 0. At this time, the second expected image can be generated according to the control information of the above sample adjustment dimension corresponding to the conditional image generation model and the sample image.

[0291] The reward value refers to the quality change between the first expected image and the second expected image.

[0292] In the embodiment of the present application, the reward value can be calculated through a trained quality model, or given through artificial negative feedback, which is not specifically limited herein.

[0293] S104: Save the initial state, initial adjustment behavior, reward value and candidate state of this time into the memory pool.

[0294] That is, save s, a (indicating the initial adjustment behavior), r, s` into the memory pool.

[0295] It should be noted that, in the present application, before step S104, it is necessary to initialize the memory pool and set a preset observation value. The preset observation value can be set based on experience, representing a threshold value of the amount of data in the memory pool. When the threshold value is exceeded, it means that the amount of data in the memory pool is sufficient, and a portion of data can be randomly extracted from these amounts to construct a training sample set.

[0296] S105: Determine whether the amount of data in the memory pool reaches a preset observation value, if yes, execute S106, otherwise, execute S107.

[0297] S106: Randomly select some data from the memory pool to construct a training sample set.

[0298] Specifically, when constructing a training sample set, s` in each selected data (s, a, r, s`) can be used as a candidate state in the training sample, and the target_q in the s` state is calculated according to the r value in each data (s, a, r, s`). The specific calculation method can be found in the above formula 1, and the repeated parts will not be repeated.

[0299] S107: Determine whether the reward value meets the preset threshold condition. If yes, return to S101; otherwise, execute S108.

[0300] S108: Take the candidate state as the initial state and return to S102.

[0301] That is, if the amount of data in the memory pool does not reach the preset observation value and the reward value meets the preset threshold condition, the learning process is terminated; if the amount of data in the memory pool does not reach the preset observation value and the reward value does not meet the preset threshold condition, the learning process is continued, the current candidate state is reset as the initial state, and the process returns to the step of selecting the initial adjustment behavior based on the initial state.

[0302] In the embodiment of the present application, the parameter done can also be used to indicate whether the current learning process is finished. In the case of finishing the current learning process, done can be recorded as 1, and in the case of not finishing the current learning process, done can be recorded as 0. In this way, the done parameter can be stored in the memory pool together with the above s, a, r, s`, and each group of s, a, r, s` corresponds to a done value.

[0303] It should be noted that for this scenario, the state space is large and the Q-table is very large. This application solves the above problem by implementing the q-table through a neural network. Input state and output the q value of different actions. Neural networks usually use nonlinear action value function approximators to express Q-functions.

[0304] In the above implementation, a sample image set and a sample adjustment dimension corresponding to the sample image are used as sample data, and AI is used to learn from the environment by interacting with the environment (through repeated trials) and receiving rewards (negative or positive) as feedback for performing actions (adjusting Lora weight values). This method can automatically learn the correlation between the adjustment dimension and the Lora weight, which is accurate and efficient.

[0305] After the weight prediction model is trained in the above manner, the lora weight set corresponding to each reference object in the reference image can be obtained based on the trained weight prediction model, and based on this, the weight matrix of each denoising network layer in the pre-trained image generation model is adjusted to obtain the trained image generation model.

[0306] On this basis, based on the trained image generation model, combined with the mask areas and control information corresponding to each reference object, the image part in the corresponding mask area can be denoised to obtain the final conditional image that meets the expectations.

[0307] See also Fig.11 As shown in FIG. 1 , it is a flow chart of conditional image generation in an embodiment of the present application. An optional implementation is to follow the steps below: Fig.11 The flowchart shown implements S24, including the following steps S111 to S115:

[0308] S111: Input a reference image into a trained image generation model, and encode the reference image based on a coding layer in the trained image generation model to obtain a coded image.

[0309] S112: Randomly add noise to the encoded image to obtain a noisy image.

[0310] S113: Encode the control information through a conditional network in the trained image generation model to obtain conditional coding features.

[0311] S114: Input the conditional coding features and the noisy image into the denoising network in the trained image generation model, perform iterative denoising processing in combination with the mask area, and obtain a denoised image.

[0312] S115: Based on the decoding layer in the trained image generation model, the denoised image is decoded to obtain a conditional image.

[0313] See also Fig.12 As shown, it is a schematic diagram of the overall framework of an image generation model in an embodiment of the present application. The basic idea of ​​the image generation model is to generate an image by iterating "denoising" data in a potential representation space, and then decode the representation result into a complete image.

[0314] in, Fig.12 The image generation model listed uses Stable Diffusion, which is an artificial intelligence image generation tool. It is a very powerful generative AI that can freely create paintings through text-generated images or image-generated images. Among them, x is the original image, that is, the reference image, is the generated image, i.e., the conditional image.

[0315] Specifically, Fig.12 The image generation model listed is a latent diffusion model (Latent Diffusion Models), which mainly includes three parts: 121 is a pixel space (Piexl Space) part, 122 is a latent representation space (Latent Space) part, and 123 is a conditional (Conditioning) part.

[0316] Among them, the 121 part is mainly an autoencoder (AutoEncoder, AE), including an encoder E (i.e. Fig.12 1211 in) and a decoder D (i.e. Fig.12 1212 in).

[0317] Specifically, AE is a pre-trained autoencoder. The purpose of the autoencoder is data dimensionality reduction. Its optimization goal is to compress the data through the encoder and then restore the data through the decoder, so that the input and output data are as similar as possible. In the embodiment of the present application, the decoder restores the data as a function of a generator, and combines the control information for the conditions of this conditional image generation task to generate the expected image.

[0318] Part 122 is mainly about the diffusion process (i.e. Fig.121221 in) and the denoising network (Denoising Unet, which can be abbreviated as Unet) (i.e. Fig.12 1222), the denoising network can also be recorded as ∈ θ The 123 part is mainly used to encode the conditions. The conditions in this application can be the control information corresponding to each reference object. DiffusionProcess is the forward process, which randomly adds noise. Denoising Unet is the reverse process, the purpose of which is to gradually denoise the distribution of random noise to generate real samples.

[0319] In the case of Stable Diffusion fine-tuning, LoRa can be applied with a cross-attention layer that links image representations with the cues that describe them. Fig.12 The QKV block in 1222 is only responsible for building the relationship between the image and text representation.

[0320] In the embodiment of the present application, the encoder 1211 can be used to compress and encode the input reference image x to obtain the encoded image z. Then, a diffusion operation is performed on the latent representation space to obtain the noise image z. T .

[0321] Among them, the diffusion operation introduces the conditional mechanism (Conditioning Mechanisms), which realizes multimodal training through cross-attention, so that the conditional image generation task can also be achieved.

[0322] The control information in the embodiment of the present application can be recorded as y. Specifically, the image generation model in the embodiment of the present application is realized by adding a cross-attention mechanism to the UNet backbone network. θ (z t , t, y), that is, to achieve the modeling of the relationship between control information and image. In addition, in order to be able to pre-process y from multiple different modalities, a domain specific encoder is introduced, that is, Fig.12 τ in θ , which is used to map y into an intermediate representation This makes it easy to introduce various forms of conditions, such as Fig.12The listed forms include semantic maps, texts, descriptions, and images. The final model can integrate the control information into the middle layer of Unet through cross-attention layer mapping. Specifically, the process of denoising in Unet with control information requires T iterations, such as the first iteration: z T becomes z T-1 , second iteration: z T-1 becomes z T-2 ,…, and so on.

[0323] Finally, use the decoder to restore it to the original pixel space. This method is called perceptual compression.

[0324] The main process of performing diffusion operation on the latent representation space is not much different from the standard diffusion model, and the diffusion model used can be a time-conditional network (time-conditional Unet). That is, in S112, noise can be randomly added to the encoded image through the time-conditional Unet to obtain a noise image.

[0325] In addition, other diffusion models may also be used, which are not specifically limited in this article.

[0326] In the above implementation, a diffusion model is used as an image generation model. The idea of ​​adding noise and denoising is adopted and conditional restrictions are introduced to realize the generation of conditional images. The information attenuation caused by noise is learned and then the learned pattern is used to generate images. Since each point on the normal distribution is a mapping of real data, the model has better interpretability, is effective and easy to implement, and can produce images of excellent quality.

[0327] Moreover, in this process, the cross-attention layer is used as the denoising network layer. The cross-attention layer allows the model to learn the relationship between different input data and better model the connection between the image and control information, so as to better understand the data and improve the accuracy of the generated conditional image.

[0328] See also Fig.13A As shown, it is a schematic diagram of multi-LORA automatic merging in an embodiment of the present application. In an embodiment of the present application, the learned LORA weight set can be directly stored as a LORA model. If there is a conditional image generation task with the same or similar requirements in the future, the corresponding LORA model can be directly loaded.

[0329] Taking the two reference objects in the reference image involved in this conditional image generation task as an example, model A can be understood as the lora model of one of the reference objects, that is, the lora weight set corresponding to the reference object, and model B can be understood as the lora model of the other reference object.

[0330] exist Fig.13A The lora weights of model A in 25 denoising network layers are listed in the figure. The input part has 12 layers, namely IN00-weight is 1, IN01-weight is 0.91666666666, IN02-weight is 0.8333333333, IN03-weight is 0.75, IN04-weight is 0.6666666666, IN05-weight is 0.5833333333, IN06-weight is 0.5, IN07-weight is 0.4166666666, IN08-weight is 0.333333333, IN09-weight is 0.25, IN10-weight is 0.1666666666, IN11-weight is 0.0833333333. There is 1 layer in the middle part, which is M00-weight is 0. The output part also has 12 layers, namely OUT00-weight 1, OUT01-weight 0.91666666666, OUT02-weight 0.8333333333, OUT03-weight 0.75, OUT04-weight 0.6666666666, OUT05-weight 0.5833333333, OUT06-weight 0.5, OUT07-weight 0.4166666666, OUT08-weight 0.333333333, OUT09-weight 0.25, OUT10-weight 0.1666666666, OUT11-weight 0.0833333333.

[0331] On this basis, by sliding, you can further view the lora weight set corresponding to model B. And after merging model A and model B (i.e. weighted summation), the lora weight set of the output model is obtained, such as Fig. 13B shown.

[0332] See also Fig. 13B As shown, it is another schematic diagram of automatic merging of multiple LoRa in an embodiment of the present application.

[0333] in, Fig. 13BTake the automatic merging method of weighted average as an example, that is, the lora weight of the output model is the weighted average of model A and model B. Assuming that the lora weights corresponding to model A and model B are the same, the comprehensive lora weight obtained by weighted average is also the same as that of model A and model B, which is still the above-listed IN00-weight is 1, IN01-weight is 0.9166666666, IN02-weight is 0.8333333333, IN03-weight is 0.75, IN04-weight is 0.666666666, IN05-weight is 0.5833333333, IN06-weight is 0.5, IN07-weight is 0.4166666666, IN08-weight is 0.333333333, IN09-weight is 0.25, IN10-weight is 0.1666666666, IN11-weight is 0.0833333333M00-weight is 0, OUT00-weight is 1, OUT01-weight is 0.9166666666, OUT02-weight is 0.8333333333, OUT03-weight is 0.75, OUT04-weight is 0.6666666666, OUT05-weight is 0.5833333333, OUT06-weight is 0.5, OUT07-weight is 0.4166666666, OUT08-weight is 0.333333333, OUT09-weight is 0.25, OUT10-weight is 0.1666666666, OUT11-weight is 0.0833333333.

[0334] It should be noted that the automatic merging of multiple Loras listed above is only a simple example, and other merging methods are also applicable to the embodiments of the present application, which are not specifically limited in this article.

[0335] See also Fig.14 As shown, it is a schematic diagram of a plurality of lora superposition results in an embodiment of the present application. Exemplary explanations are given from the dimensions of facial details, expressions, clothing, decoration, posture (divided into posture control and posture label), background and style. Among them, posture control and posture label are two ways to define posture. Posture control is specifically defined by a skeleton diagram, while posture label is defined by a tag text label. It can also be defined in other ways, which is not specifically limited in this article.

[0336] In summary, this application avoids conflicts caused by LoRa fusion by adjusting the weights of conflicting layers and controlling the influence range of LoRa. In addition, this application uses a hierarchical control method to remove overfitting parts. This application can automatically search for the optimal weight combination result based on reinforcement learning, which is more efficient and accurate.

[0337] Based on the same inventive concept, the present application also provides an image generating device. Fig.15 As shown, it is a schematic diagram of the structure of the image generating device 1500, which may include:

[0338] The data acquisition unit 1501 is used to acquire a reference image and control information of each reference object in the reference image, where the control information indicates: an adjustment condition of the corresponding reference object in at least one specified adjustment dimension;

[0339] The weight prediction unit 1502 is used to perform the following operations for each reference object: based on the control information associated with a reference object, obtain a low-rank adaptive weight set; the low-rank adaptive weight set includes: the low-rank adaptive weights of each denoising network layer included in the pre-trained image generation model, wherein each denoising network layer is associated with at least one adjustment dimension;

[0340] A model adjustment unit 1503 is used to adjust the network parameters of each denoising network layer in the pre-trained image generation model based on each low-rank adaptive weight set obtained to obtain a trained image generation model;

[0341] The image generation unit 1504 is used to input the reference image and the obtained control information into the trained image generation model to obtain a conditional image generated by adjusting the reference image in combination with the respective mask areas of the reference objects; each mask area is determined based on the position information of the corresponding reference object in the reference image.

[0342] Optionally, the model adjustment unit 1503 is specifically configured to:

[0343] For each denoising network layer, perform the following operations:

[0344] Perform weighted summation on the low-rank adaptive weights of each reference object for a denoising network layer to obtain a comprehensive low-rank adaptive weight of the denoising network layer;

[0345] Adjusting a weight matrix of a denoising network layer in a pre-trained image generation model based on a comprehensive low-rank adaptive weight of the denoising network layer to obtain a corresponding adjusted weight matrix;

[0346] The trained image generation model is determined according to the adjusted weight matrices of each denoising network layer in the pre-trained image generation model.

[0347] Optionally, the model adjustment unit 1503 is further configured to:

[0348] Based on the obtained low-rank adaptive weight sets, the network parameters of each denoising network layer in the pre-trained image generation model are adjusted, and before the trained image generation model is obtained, initial weights set for each reference object are obtained, each initial weight is positively correlated with the similarity between the corresponding reference object and the expected adjustment result;

[0349] The model adjustment unit 1503 is specifically used for:

[0350] Based on the initial weights of each reference object, the low-rank adaptive weights of a denoising network layer are weighted summed to obtain a comprehensive low-rank adaptive weight of the denoising network layer.

[0351] Optionally, the model adjustment unit 1503 is specifically configured to:

[0352] For each denoising network layer, the following operations are performed respectively: according to the low-rank adaptive weights of each reference object for each denoising network layer, a weight matrix corresponding to a denoising network layer in the pre-trained image generation model is adjusted to obtain a corresponding adjusted weight matrix;

[0353] The trained image generation model is determined according to the adjusted weight matrices of each denoising network layer in the pre-trained image generation model.

[0354] Optionally, the weight prediction unit 1502 is specifically used for:

[0355] Inputting control information associated with a reference object into a trained weight prediction model to obtain a low-rank adaptive weight set output by the trained weight prediction model;

[0356] Among them, the trained weight prediction model is obtained through reinforcement learning training.

[0357] Optionally, the device is also used for:

[0358] The model training unit 1505 is used to obtain the trained weight prediction model by training in the following manner:

[0359] A training sample set is obtained, where each training sample in the training sample set includes: a candidate state, and the true value of the long-term return corresponding to each candidate adjustment behavior under the candidate state; the candidate state represents: a candidate low-rank adaptive weight for each denoising network layer in the pre-trained image generation model; each candidate adjustment behavior represents: an adjustment amount when adjusting the candidate low-rank adaptive weight;

[0360] Based on the training sample set, the weight prediction model to be trained is subjected to cyclic iterative training to obtain a trained weight prediction model, wherein each iterative training performs the following process:

[0361] Select training samples from the training sample set, and input the candidate states in the training samples into the weight prediction model to be trained to obtain the long-term return estimation value corresponding to each adjustment behavior under the candidate state;

[0362] Based on the difference between each long-term return estimate and the corresponding true value of the long-term return, the parameters of the weight prediction model to be trained are adjusted.

[0363] Optionally, the model training unit 1505 is specifically used for:

[0364] The following learning process is iteratively performed based on the sample image set until the amount of data in the memory pool reaches the preset observation value, and then a portion of the data is randomly selected from the memory pool to construct a training sample set; wherein each learning process performs the following operations:

[0365] A sample image is selected from the sample image set, and an initial state is set according to the sample adjustment dimension of the sample image, where the initial state represents: the initial low-rank adaptive weights of each denoising network layer in the pre-trained image generation model;

[0366] Selecting an initial adjustment behavior according to the initial state, the initial adjustment behavior represents: an adjustment amount when adjusting the initial low-rank adaptive weight;

[0367] After performing an initial adjustment on the pre-trained image generation model, a reward value and a candidate state after performing the initial adjustment are obtained; the reward value represents: a change in quality of the image generated by the image generation model before and after the adjustment;

[0368] Save the initial state, initial adjustment behavior, reward value and candidate state to the memory pool.

[0369] Optionally, the model training unit 1505 is further used for:

[0370] In each learning process, if the reward value meets the preset threshold condition, the learning process ends;

[0371] If the reward value does not meet the preset threshold condition, the candidate state is used as the initial state, and the process returns to the step of selecting the initial adjustment behavior according to the initial state.

[0372] Optionally, the denoising network layer is a cross-attention layer included in the denoising network in the image generation model; the cross-attention layer is used to model the connection between the control information and the image.

[0373] Optionally, the image generating unit 1504 is specifically configured to:

[0374] Inputting a reference image into a trained image generation model, encoding the reference image based on a coding layer in the trained image generation model to obtain a coded image;

[0375] Randomly add noise to the encoded image to obtain a noisy image;

[0376] Encode the control information through the conditional network in the trained image generation model to obtain conditional encoding features;

[0377] Input the conditional coding features and the noisy image into the denoising network in the trained image generation model, perform iterative denoising processing on the masked area, and obtain a denoised image;

[0378] Based on the decoding layer in the trained image generation model, the denoised image is decoded to obtain a conditional image.

[0379] In summary, the present application does not simply fuse multiple LoRas, but controls different denoising network layers through the hierarchical setting of LoRa weights to achieve control of different dimensions of the image. Specifically, after obtaining the control information of the reference image and each reference object in the reference image, the present application takes into account that each of these control information corresponds to at least one specified adjustment dimension. Therefore, based on these control information, a low-rank adaptive weight set for the pre-trained image generation model can be determined, and the weight set includes the low-rank adaptive weights of each denoising network layer contained in the pre-trained image generation model, so as to hierarchically lock the LoRa weights and avoid conflicts between LoRas. The trained image generation model obtained on this basis can achieve control of different dimensions of the image. In addition, when the present application generates a conditional image based on the generated trained image generation model, it further introduces the feature of the mask area related to the reference object, and determines the scope of action of LoRa through the mask area. On the basis of LoRa stratification, the effect of LoRa regional control is further achieved, avoiding conflicts between LoRas and improving the accuracy of the generated conditional image.

[0380] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0381] After introducing the image generating method and apparatus according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0382] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0383] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.16 As shown, it includes a memory 1601 , a communication module 1603 and one or more processors 1602 .

[0384] The memory 1601 is used to store computer programs executed by the processor 1602. The memory 1601 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0385] The memory 1601 may be a volatile memory, such as a random-access memory (RAM); the memory 1601 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1601 is any other medium that can be used to carry or store an expected computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1601 may be a combination of the above memories.

[0386] The processor 1602 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1602 is used to implement the above-mentioned image generation method when calling the computer program stored in the memory 1601 .

[0387] The communication module 1603 is used to communicate with terminal devices and other servers.

[0388] The specific connection medium between the memory 1601, the communication module 1603 and the processor 1602 is not limited in the embodiment of the present application. Fig.16 In the embodiment, the memory 1601 and the processor 1602 are connected via a bus 1604. The bus 1604 is connected to the processor 1602 via a bus 1604. Fig.16 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1604 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.16 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0389] The memory 1601 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the image generation method of the embodiment of the present application. The processor 1602 is used to execute the above-mentioned image generation method, such as Figure 2 shown.

[0390] In another embodiment, the electronic device may also be other electronic devices, such as Figure 1 The terminal device 110 shown in FIG. 1 is a terminal device 110 shown in FIG. 1 . In this embodiment, the structure of the electronic device can be as follows: Fig.17 As shown, it includes: a communication component 1710, a memory 1720, a display unit 1730, a camera 1740, a sensor 1750, an audio circuit 1760, a Bluetooth module 1770, a processor 1780 and other components.

[0391] The communication component 1710 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the user to send and receive information through the WiFi module.

[0392] The memory 1720 can be used to store software programs and data. The processor 1780 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1720. The memory 1720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The memory 1720 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 1720 can store an operating system and various application programs, and can also store a computer program for executing the image generation method of the embodiment of the present application.

[0393] The display unit 1730 may also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1730 may include a display screen 1732 disposed on the front of the terminal device 110. The display screen 1732 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 1730 may be used to display a setting interface or other user interfaces of the image production platform in the embodiment of the present application.

[0394] The display unit 1730 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device 110. Specifically, the display unit 1730 may include a touch screen 1731 arranged on the front of the terminal device 110, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0395] The touch screen 1731 can be covered on the display screen 1732, or the touch screen 1731 and the display screen 1732 can be integrated to realize the input and output functions of the terminal device 110, and the integrated display screen can be referred to as a touch display screen. In this application, the display unit 1730 can display the application and the corresponding operation steps.

[0396] Camera 1740 can be used to capture static images, and users can publish images taken by camera 1740 through the application. Camera 1740 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the processor 1780 to convert it into a digital image signal.

[0397] The terminal device may further include at least one sensor 1750, such as an acceleration sensor 1751, a distance sensor 1752, a fingerprint sensor 1753, and a temperature sensor 1754. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0398] The audio circuit 1760, the speaker 1761, and the microphone 1762 can provide an audio interface between the user and the terminal device 110. The audio circuit 1760 can transmit the electrical signal converted from the received audio data to the speaker 1761, which is converted into a sound signal for output. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1760 and converted into audio data, and then the audio data is output to the communication component 1710 to be sent to, for example, another terminal device 110, or the audio data is output to the memory 1720 for further processing.

[0399] The Bluetooth module 1770 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1770 to exchange data.

[0400] The processor 1780 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1720 and calling data stored in the memory 1720. In some embodiments, the processor 1780 may include one or more processing units; the processor 1780 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1780. In the present application, the processor 1780 can run the operating system, application programs, user interface display and touch response, as well as the image generation method of the embodiment of the present application. In addition, the processor 1780 is coupled to the display unit 1730.

[0401] In some possible implementations, various aspects of the image generation method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the image generation method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 2 Follow the steps shown in .

[0402] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0403] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0404] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0405] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0406] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0407] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0408] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the expected results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0409] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0410] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0411] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0412] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0413] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0414] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a reference image and control information of each reference object in the reference image, wherein the control information indicates: an adjustment condition of the corresponding reference object in at least one specified adjustment dimension; For each reference object, the following operations are performed respectively: based on control information associated with a reference object, a low-rank adaptive weight set is obtained; the low-rank adaptive weight set includes: low-rank adaptive weights of each denoising network layer included in the pre-trained image generation model, wherein each denoising network layer is associated with at least one adjustment dimension; Based on the obtained low-rank adaptive weight sets, adjusting the network parameters of each denoising network layer in the pre-trained image generation model to obtain a trained image generation model; The reference image and the obtained control information are input into the trained image generation model to obtain a conditional image generated by adjusting the reference image in combination with the respective mask areas of the reference objects; each mask area is determined based on the position information of the corresponding reference object in the reference image.

2. The method according to claim 1, characterized in that The method of adjusting the network parameters of each denoising network layer in the pre-trained image generation model based on each low-rank adaptive weight set obtained to obtain the trained image generation model includes: For each denoising network layer, perform the following operations: Performing weighted summation on the low-rank adaptive weights of each reference object for a denoising network layer to obtain a comprehensive low-rank adaptive weight of the denoising network layer; Based on the comprehensive low-rank adaptive weight of the denoising network layer, adjust the weight matrix of the denoising network layer in the pre-trained image generation model to obtain the corresponding adjusted weight matrix; determine the trained image generation model according to the adjusted weight matrices of each denoising network layer in the pre-trained image generation model.

3. The method according to claim 2, characterized in that Before adjusting the network parameters of each denoising network layer in the pre-trained image generation model based on the obtained low-rank adaptive weight sets to obtain the trained image generation model, the method further includes: Acquire initial weights set for each reference object, each initial weight being positively correlated with a similarity between the corresponding reference object and the expected adjustment result; The step of weighting and summing the low-rank adaptive weights of each reference object for a denoising network layer to obtain a comprehensive low-rank adaptive weight of the denoising network layer includes: Based on the initial weights of the reference objects, the low-rank adaptive weights of the denoising network layer are weighted summed to obtain a comprehensive low-rank adaptive weight of the denoising network layer.

4. The method according to claim 1, characterized in that The method of adjusting the network parameters of each denoising network layer in the pre-trained image generation model based on each low-rank adaptive weight set obtained to obtain the trained image generation model includes: For each denoising network layer, the following operations are respectively performed: adjusting a weight matrix corresponding to the denoising network layer in the pre-trained image generation model according to the low-rank adaptive weights of each reference object for the denoising network layer, to obtain a corresponding adjusted weight matrix; The trained image generation model is determined according to the adjusted weight matrices of each denoising network layer in the pre-trained image generation model.

5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining a low-rank adaptive weight set based on control information associated with a reference object includes: Inputting the control information associated with the reference object into a trained weight prediction model to obtain a low-rank adaptive weight set output by the trained weight prediction model; Wherein, the trained weight prediction model is obtained through reinforcement learning training.

6. The method according to claim 5, characterized in that The trained weight prediction model is obtained by training in the following manner: Obtain a training sample set, wherein each training sample in the training sample set includes: a candidate state, and a true value of a long-term reward corresponding to each candidate adjustment behavior under the candidate state; the candidate state represents: a candidate low-rank adaptive weight for each denoising network layer in the pre-trained image generation model; each candidate adjustment behavior represents: an adjustment amount when adjusting the candidate low-rank adaptive weight; Based on the training sample set, a weight prediction model to be trained is subjected to cyclic iterative training to obtain the trained weight prediction model, wherein each iterative training performs the following process: Selecting training samples from the training sample set, and inputting candidate states in the training samples into the weight prediction model to be trained, to obtain the long-term return estimation value corresponding to each adjustment behavior under the candidate state; Based on the difference between each obtained long-term reward estimate and the corresponding long-term reward true value, the parameters of the weight prediction model to be trained are adjusted.

7. The method according to claim 6, characterized in that The step of obtaining a training sample set includes: The following learning process is iteratively performed based on the sample image set until the amount of data in the memory pool reaches a preset observation value, and then a portion of data is randomly selected from the memory pool to construct the training sample set; wherein each learning process performs the following operations: Selecting a sample image from the sample image set, and setting an initial state according to a sample adjustment dimension of the sample image, wherein the initial state represents: initial low-rank adaptive weights of each denoising network layer in the pre-trained image generation model; Selecting an initial adjustment behavior according to the initial state, the initial adjustment behavior representing: an adjustment amount when adjusting the initial low-rank adaptive weight; After performing the initial adjustment behavior on the pre-trained image generation model, a reward value and a candidate state after performing the initial adjustment behavior are obtained; the reward value represents: a quality change of an image generated by the image generation model before and after the adjustment; Save the initial state, initial adjustment behavior, reward value and candidate state to the memory pool.

8. The method according to claim 7, characterized in that During each learning process, the method further comprises: If the reward value meets the preset threshold condition, the learning process ends; If the reward value does not meet the preset threshold condition, the candidate state is used as the initial state, and the process returns to the step of selecting an initial adjustment behavior according to the initial state.

9. The method according to any one of claims 1 to 4, characterized in that: The denoising network layer is a cross-attention layer included in the denoising network in the image generation model; the cross-attention layer is used to model the connection between control information and the image.

10. The method according to claim 9, characterized in that The step of inputting the reference image and each obtained control information into the trained image generation model to obtain a conditional image generated by adjusting the reference image in combination with the respective mask regions of each reference object comprises: Inputting the reference image into the trained image generation model, encoding the reference image based on the encoding layer in the trained image generation model to obtain an encoded image; Randomly adding noise to the coded image to obtain a noisy image; Encoding the control information through a conditional network in the trained image generation model to obtain conditional coding features; Inputting the conditional coding features and the noise image into the denoising network in the trained image generation model, performing iterative denoising processing in combination with the mask region, and obtaining a denoised image; Based on the decoding layer in the trained image generation model, the denoised image is decoded to obtain the conditional image.

11. An image generating device, characterized in that: include: A data acquisition unit, configured to acquire a reference image and control information of each reference object in the reference image, wherein the control information indicates an adjustment condition of the corresponding reference object under at least one specified adjustment dimension; The weight prediction unit is used to perform the following operations for each reference object: obtaining a low-rank adaptive weight set based on control information associated with a reference object; The low-rank adaptive weight set includes: low-rank adaptive weights of each denoising network layer included in the pre-trained image generation model, wherein each denoising network layer is associated with at least one adjustment dimension; A model adjustment unit, configured to adjust network parameters of each denoising network layer in the pre-trained image generation model based on each low-rank adaptive weight set obtained, so as to obtain a trained image generation model; An image generation unit is used to input the reference image and the obtained control information into the trained image generation model to obtain a conditional image generated by adjusting the reference image in combination with the respective mask areas of the reference objects; each mask area is determined based on the position information of the corresponding reference object in the reference image.

12. The device according to claim 11, characterized in that The denoising network layer is a cross-attention layer included in the denoising network in the image generation model; the cross-attention layer is used to model the connection between control information and the image.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The method comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the methods of claims 1 to 11.

15. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the methods described in claims 1 to 11.

Images