A method and system for image editing based on multimodal condition adaptation

Abstract

This invention provides an image editing method and system based on multimodal condition adaptation. The image editing method includes: a first text vector acquisition step: processing to obtain an image description corresponding to the image, processing the image description with an editor to obtain a first text vector based on the image information; a second text vector acquisition step: processing an editing instruction to obtain a second text vector based on the editing instruction; a fusion feature acquisition step: fusing the first and second text vectors through weights to obtain fusion features; an image latent feature encoding acquisition step; an image editing step: receiving injected conditional information, simultaneously denoising the received image latent features and latent noise, with the received conditional information guiding the iterative denoising process to gradually generate the image desired by the user; and an image restoration step. This invention can improve the stability, controllability, and accuracy of image editing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image editing method and system based on multimodal condition adaptation. Background Technology

[0002] Traditional image editing algorithms mainly fall into two categories:

[0003] I. The basic pixel operation algorithm reads the pixel matrix of the image, iterates through each pixel and applies the corresponding calculation rules according to the set adjustment parameters, and after the calculation is completed, updates the pixel matrix and saves the edited image;

[0004] II. Filter-based algorithms utilize filters to perform convolution operations on images, achieving effects such as image smoothing, sharpening, and edge detection. By performing convolution calculations on each pixel position of the image, a filtered image is obtained, thereby achieving the corresponding editing effects.

[0005] Traditional image editing algorithms struggle to achieve ideal results for complex image editing tasks, such as semantic understanding and advanced style transfer. Pixel-based algorithms have limited capabilities when processing global semantic information; for example, they struggle to accurately identify and replace specific objects when performing intelligent content replacement. While filtering-based algorithms may suffer from blurring or distortion when handling complex textures and structures—for instance, over-filtering richly detailed natural scene images can lead to the loss of important details—they are typically based on fixed rules and parameters, lacking adaptability to different image content and editing needs. For example, when adjusting image contrast, a fixed contrast adjustment algorithm may not be optimized for different lighting conditions and image subjects, leading to over-enhancing or weakening contrast in some cases, affecting image quality. Moreover, when image content changes significantly, it may require redesigning the algorithm or adjusting parameters, resulting in poor flexibility.

[0006] Currently popular image editing techniques are based on deep learning algorithms, including those based on GANs (Generative Adversarial Networks). These algorithms train GANs on large-scale image datasets to achieve a certain performance level for the generator and discriminator. During the editing phase, the generator's input is adjusted according to the editing requirements. For example, to perform style transfer, the style features of the source image can be extracted and combined with a random noise vector as the generator's input to generate an image with the target style; alternatively, image deformation and modification can be achieved by adjusting certain parameters in the generator network, and the generator produces the edited image based on the new input. Diffusion model-based methods first train the diffusion model on a large amount of image data, learning the parameters during forward and backward diffusion processes. During editing, based on editing instructions (such as "change the color of objects in the image to red"), the model is guided to generate the edited image by adjusting the noise addition method during forward diffusion or by utilizing conditional information during backward diffusion. For example, the DDIM Inverion module generates a new image from the reconstructed input image under the guidance of new text. However, the reconstruction and regeneration process is unstable, leading to inconsistencies between the edited image and the input image or target description. Another approach involves user-instructed editing and attention control, adjusting the generated image by controlling attention, but this has very limited data type matching and is difficult to generalize. Alternatively, a data-driven approach could be used, preparing a large-scale pairwise edit dataset to train the instruction diffusion model. However, the instability of the reconstruction and generation process in these methods can lead to errors in the edited image, resulting in inconsistencies with the input image and user editing instructions. Furthermore, it is difficult to collect high-quality datasets covering all types of edit pairs, and the data creation tools used are inherently inadequate, limiting model performance.

[0007] In summary, existing image editing technologies have the following problems:

[0008] 1. Diverse editing instructions and image descriptions concatenated together can lead to text loss. When a detailed image description is combined with user editing instructions, the text becomes too long and is truncated when it enters a text encoder. At this point, the model may misunderstand the image description and the area that the user actually wants to edit.

[0009] 2. Insufficient ability to handle severe image degradation. Most image generation is not trained on low-level images, so relying on instructions may not yield ideal results when image degradation is severe.

[0010] 3. Lack of stability: Instability during reconstruction and regeneration makes the edited image prone to errors and inconsistent with the input image or target description;

[0011] 4. Lack of controllability: Although the generation can be guided by input conditions, the generated results still have a certain degree of randomness, making it difficult to precisely control every detail of the image;

[0012] Therefore, it is necessary to propose an optimized image editing algorithm that allows users to edit images freely with just one sentence and obtain the desired image results. Summary of the Invention

[0013] To address the problem that image editing in existing technologies cannot achieve ideal results, this invention provides an image editing method and system based on multimodal condition adaptation.

[0014] The present invention provides an image editing method based on multimodal condition adaptation, comprising the following steps:

[0015] The first text vector acquisition steps are as follows: acquire the input image, process it to obtain the image description corresponding to the image, process the image description through an editor, and obtain the first text vector based on the image information;

[0016] The second text vector acquisition step is to obtain the user's editing instructions for the image, process the editing instructions, and obtain the second text vector based on the editing instructions.

[0017] The fusion feature acquisition step is as follows: weights are assigned to the first text vector and the second text vector to obtain fusion features based on the assigned weights and by fusing the first text vector and the second text vector;

[0018] Image latent feature encoding acquisition steps: The input image is processed by an image encoder and compressed into the latent space to obtain the image latent features;

[0019] Image editing steps: The diffusion model receives conditional information injected through a cross-attention mechanism. The conditional information includes a first text vector, a second text vector, and fused feature information. At the same time, the diffusion model denoises the latent features and latent noise of the received image. Guided by the received conditional information, the image desired by the user is gradually generated during the iterative denoising process.

[0020] Image restoration steps: The edited image is restored to a pixel-space image using an image decoder.

[0021] Furthermore, in the first text vector acquisition step, the image is converted into an image description using a Visual Language Model (VLM). The VLM model, through pre-training, establishes a mapping relationship between image content and natural language description. The training method for the VLM is as follows:

[0022] A1: The image encoder and text encoder of the VLM model extract features from the image and text, respectively;

[0023] A2: The image encoder converts an image into a visual feature vector v, and the text encoder converts text into a text feature vector t;

[0024] A3: Calculate the similarity score S between positive sample pairs. pos =vTt pos And the similarity score between negative sample pairs. neg =v T t neg , where t pos t is the feature vector of the positive sample text. neg It is the feature vector of the negative sample text, v T This represents the transpose of vector v;

[0025] A4: Calculate the loss L by comparing the learning loss function. contrastive The calculation formula is: Where N is the number of negative samples, and i is the index variable for summation;

[0026] A5: Using masked language model loss, by randomly masking some words in the text, the model can predict the masked words based on the image content and the unmasked words, thereby enabling the model to learn the language structure and semantic relationships of the text;

[0027] A6: Use a text encoder to extract features from the masked text to obtain the masked feature vector t. masked ;

[0028] A7: Input this feature vector and the corresponding image feature vector v together into the decoder or prediction layer of the VLM model to predict the probability distribution p of the masked words. masked ;

[0029] A8: Calculate the loss of the masked language model using the cross-entropy loss function. For each masked word, the loss is... Among them, y kis the true label of the masked word, and M is the number of masked words. This loss function enables the model to accurately predict the probability distribution of the masked words based on the image content and context information. By optimizing the masked language model loss, the VLM model learns to generate reasonable language descriptions based on the image content.

[0030] Furthermore, after acquiring the input image, the VLM model analyzes the image and generates a natural language description to summarize the key elements in the image, including scenes, people, and / or objects.

[0031] Furthermore, in the feature fusion acquisition step, a shared attention mechanism is adopted. By interactively calculating each element in the first and second text vectors, the fusion features are obtained. The specific processing method is as follows:

[0032] Attention scores are calculated using dot product operations. Let V be the first text vector of the image description. d The second text vector of the user's editing command is V. u For the first text vector V d Each element d in j Second text vector V u For each element ux in the dataset, calculate the attention score. Attention scores reflect the degree of semantic association between [the two elements].

[0033] By normalizing all attention scores, the weight w of each element is obtained. j,k This weight represents the weight of each element d in the first text vector of the image description when generating fused features. j For each element u in the second text vector of the user's editing instructions k The relative importance of.

[0034] Furthermore, in the image latent feature encoding acquisition step, an image variational autoencoder (VAE) is used to convert the pixel information of the image into an intermediate feature representation. The final image latent features include a mean vector μ and a variance vector σ. The VAE encodes the image into a latent noise vector in the latent space. The latent noise vector represents the key information of the image in a compact form.

[0035] Furthermore, the original dimension of the first text vector is d1. After passing through a fully connected layer with a weight matrix W1, a transformed first vector c1′ = W1c1 is obtained, which is consistent with the dimension d2 of the latent feature vector of the image. Here, c1 is the first text vector. Similarly, a similar linear transformation is performed on the second text vector and the fused features to obtain the second vector c2′ and the third vector c3′. Then, the input image is processed by the encoder of the VAE to obtain the latent vector z of the image. This latent vector contains the key feature information of the image and exists in a compact low-dimensional form, providing a basis for fusion with conditional information.

[0036] Furthermore, the specific processing method for the image editing steps is as follows:

[0037] B1: For each element z in the latent vector z b and condition vector c′ a For each element in the set, a = 1, 2, 3, the attention score is... i represents conditional information used to identify different sources, and k is an index variable used to index the condition vector c′. a The elements in

[0038] B2: By normalizing, we obtain each element z in the latent vector z. b attention weights The attention weight represents the condition vector c′ during the editing process. a The relative importance of each element in the vector z to each element in the latent vector z;

[0039] B3: Apply the attention weights obtained in step B2 Value vector with conditional information Multiplying and summing yields the latent image features f that incorporate conditional information. n Image latent features f n The calculation formula is:

[0040]

[0041] in, Represents the condition vector c′ a Each element in the latent vector z corresponds to an element z in the latent vector z. b The relative importance of c a k is a pair of elements z in the latent vector z b The relative importance parameter, Let c′ be the condition vector a element c a The vector of values ​​for k;

[0042] B4: Denoising is performed on the latent image features and random latent noise obtained in the latent image feature encoding step. During the denoising process, the diffusion model receives guidance from conditional information and gradually adjusts the latent noise vector to change in a direction that conforms to the user's editing intention.

[0043] In the forward diffusion process of the diffusion model, noise is gradually added to the initial latent feature representation of the image obtained from the image encoder. This process is represented by the following formula:

[0044]

[0045] Among them, z n,s z is the s-th potential noise vector. n,s-1 It is the potential noise vector from the previous step, a s It is a coefficient that controls the degree of noise addition, ∈ s It is Gaussian noise. The forward diffusion process causes the potential noise vector to gradually lose its original image information, eventually becoming a distribution that is close to pure noise.

[0046] In the reverse denoising process, the goal of the diffusion model is to recover the latent image features that match the user's editing intent from the latent noise vector that is overwhelmed by noise. The neural network in the diffusion model predicts the noise in each diffusion step and uses this predicted noise to update the estimated value z′ of the latent noise vector. n,s-1 The calculation formula is:

[0047]

[0048] Where, z′ n,s Let be the noise vector at diffusion step s, ∈ θ (z′ n,s ,s) is the prediction of the neural network with parameter θ in the diffusion model, corresponding to the latent noise vector z′ at diffusion step s. n,s In this process, conditional information influences noise prediction through a cross-attention mechanism, causing the update of the potential noise vector to proceed in a direction that aligns with the user's editing intent.

[0049] This invention also provides an image editing system based on multimodal condition adaptation, used in the aforementioned image editing method based on multimodal condition adaptation, comprising:

[0050] First text vector acquisition module: used to acquire the input image, process it to obtain the image description corresponding to the image, process the image description through the editor, and obtain the first text vector based on the image information;

[0051] The second text vector acquisition module is used to acquire the user's editing instructions for the image, process the editing instructions, and obtain a second text vector based on the editing instructions.

[0052] The fusion feature acquisition module is used to assign weights to the first text vector and the second text vector, and obtain fusion features based on the assigned weights and by fusing the first text vector and the second text vector;

[0053] Image latent feature encoding acquisition module: used to compress the input image into the latent space through an image encoder to obtain the image latent features;

[0054] Image editing module: Implemented through a diffusion model, it is used to receive conditional information injected through a cross-attention mechanism. The conditional information includes a first text vector, a second text vector, and fused feature information. At the same time, the diffusion model denoises the latent features and latent noise of the received image. The guidance of the received conditional information gradually generates the image desired by the user during the iterative denoising process.

[0055] Image restoration module: Used to restore edited images into pixel-space images using an image decoder.

[0056] Compared with the prior art, the beneficial effects of the present invention are: the method of the present invention can improve the stability, controllability, and accuracy of image editing; specifically,

[0057] 1. The image description obtained from the input image and the user's editing instructions are output by separate models and concatenated to avoid the problem of excessively long text and information loss caused by splicing the two together. Each text encoder can focus on effectively encoding its own text content and converting it into a text vector.

[0058] 2. Based on the diffusion model, image reconstruction and editing are performed using latent noise and multimodal conditional information. During the training process, the diffusion model can be trained with a large amount of image degradation data of different degrees, so that it learns how to recover high-quality images from latent noise.

[0059] 3. Based on multiple conditions, cross-attention is injected into the Diffusion Transformer model. These conditions guide the image editing process at different stages, ensuring that each step is under control.

[0060] 4. User editing instructions are injected into the DiffusionTransformer model through an independent text encoder and conditional information. Using a cross-attention mechanism, the model can accurately focus on the user's requested editing content. Simultaneously, the image decoder is also constrained by these conditions when generating the final image. Attached Figure Description

[0061] To more clearly illustrate the solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0062] Figure 1 This is a flowchart of the image editing method of the present invention;

[0063] Figure 2 This is an algorithm architecture diagram of an embodiment of the editing method of the present invention. Detailed Implementation

[0064] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects, not to describe a particular order.

[0065] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment to other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0066] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0067] like Figure 1 As shown, the image editing method based on multimodal condition adaptation of the present invention includes the following steps:

[0068] The first text vector acquisition steps are as follows: acquire the input image, process it to obtain the image description corresponding to the image, process the image description through an editor, and obtain the first text vector based on the image information;

[0069] The second text vector acquisition step is to obtain the user's editing instructions for the image, process the editing instructions, and obtain the second text vector based on the editing instructions.

[0070] The fusion feature acquisition step is as follows: weights are assigned to the first text vector and the second text vector to obtain fusion features based on the assigned weights and by fusing the first text vector and the second text vector;

[0071] Image latent feature encoding acquisition steps: The input image is processed by an image encoder and compressed into the latent space to obtain the image latent features;

[0072] Image editing steps: The diffusion model receives conditional information injected through a cross-attention mechanism. The conditional information includes a first text vector, a second text vector, and fused feature information. At the same time, the diffusion model denoises the latent features and latent noise of the received image. Guided by the received conditional information, the image desired by the user is gradually generated during the iterative denoising process.

[0073] Image restoration steps: The edited image is restored to a pixel-space image using an image decoder.

[0074] This invention improves the stability, controllability, and accuracy of image editing. Its main advantages are:

[0075] 1. The image description obtained from the input image and the user's editing instructions are output by separate models and concatenated to avoid the problem of excessively long text and information loss caused by splicing the two together. Each text encoder can focus on effectively encoding its own text content and converting it into a text vector.

[0076] 2. Based on the diffusion model, image reconstruction and editing are performed using latent noise and multimodal conditional information. During the training process, the diffusion model can be trained with a large amount of image degradation data of different degrees, so that it learns how to recover high-quality images from latent noise.

[0077] 3. Based on multiple conditions, cross-attention is injected into the Diffusion Transformer model. These conditions guide the image editing process at different stages, ensuring that each step is under control.

[0078] 4. User editing instructions are injected into the DiffusionTransformer model through an independent text encoder and conditional information. Using a cross-attention mechanism, the model can accurately focus on the user's requested editing content. Simultaneously, the image decoder is also constrained by these conditions when generating the final image.

[0079] like Figure 2As shown in the example, in the first text vector acquisition step, the image is converted into an image description using a VLM model. The focus of this step is on a deep understanding of the input image. Then, this image description is processed by a text encoder to obtain the first text vector of the input image, which is defined as condition 1. The VLM model is obtained through pre-training on a large amount of image-text pair data, and it can learn the mapping relationship between image content and natural language description. Preferably, in this example, during pre-training, the model improves its ability to understand image content and generate language by optimizing the objective function, such as contrastive learning loss or masked language model loss.

[0080] The training method for VLM in this example is as follows:

[0081] A1: The image encoder and text encoder of the VLM model extract features from the image and text, respectively;

[0082] A2: The image encoder converts an image into a visual feature vector v, and the text encoder converts text into a text feature vector t;

[0083] A3: Calculate the similarity score s between positive sample pairs. pos =v T t pos And the similarity score between negative sample pairs. neg =v T t neg , where t pos t is the feature vector of the positive sample text. neg It is the feature vector of the negative sample text, v T This represents the transpose of vector v;

[0084] A4: Calculate the loss L by comparing the learning loss function. contrastive The calculation formula is: Where N is the number of negative samples and i is the index variable for summation. In this example, this loss function encourages the model to make the similarity score of positive sample pairs higher than that of negative sample pairs, so that the model can learn the semantic matching relationship between the image and the correct text description.

[0085] A5: Using masked language model loss, by randomly masking some words in the text, the model can predict the masked words based on the image content and the unmasked words, thereby enabling the model to learn the language structure and semantic relationships of the text;

[0086] A6: Use a text encoder to extract features from the masked text to obtain the masked feature vector t. masked ;

[0087] A7: Input this feature vector and the corresponding image feature vector v together into the decoder or prediction layer of the VLM model to predict the probability distribution p of the masked words. masked ;

[0088] A8: Calculate the loss of the masked language model using the cross-entropy loss function. For each masked word, the loss is... Among them, y k is the true label of the masked word (in this example, the one-hot encoding of the masked word in the original text), and M is the number of masked words. This loss function enables the model to accurately predict the probability distribution of the masked words based on the image content and contextual information.

[0089] This example optimizes the masked language model loss, enabling the VLM model to learn to generate reasonable language descriptions based on image content. During the generation process, the model utilizes visual information from the image to guide language generation, ensuring that the generated text is grammatically correct, semantically reasonable, and closely related to the image content, thereby improving the accuracy of the converted image descriptions.

[0090] This example feeds the input image into a Visual Model (VLM). The VLM model analyzes the image and generates a natural language description summarizing key elements such as the scene, people, and objects in the image. For example, for an image containing a woman smiling on grass, the VLM might generate a description like "A woman is sitting on the grass, smiling." Then, the image description generated by the VLM is input into a text encoder, resulting in the first text vector of the input image, defined as condition 1. After the text encoder converts the image description into a vector, it facilitates subsequent fusion and processing with other condition vectors, ensuring information consistency and operability throughout the entire image editing workflow.

[0091] The key step in the second text vector acquisition process is to input the user's editing instructions into the text encoder to obtain a second text vector of the user's editing instructions, which serves as condition 2. Once the user's editing instructions are converted into a second text vector, they can be easily fused and interacted with other condition vectors during subsequent image editing. This vector representation ensures that the semantic information of the editing instructions is accurately transmitted and applied throughout the entire editing process, enabling image editing operations to be performed according to the user's intentions, thus improving the accuracy and controllability of the editing.

[0092] In the feature fusion acquisition step, the focus is on fusing the features of the two text vectors from the previous two steps as condition 3. The shared attention mechanism aims to simultaneously focus on information from two or more different sources and assign weights based on their relevance. The relationship between the text vector from the image description and the text vector from the user's editing instruction is analyzed using shared attention. For example, if the image description mentions a person's clothing, and the user's editing instruction is about changing the color of that person's clothing, the shared attention mechanism will identify the semantic relevance of these two vectors regarding the person's clothing and assign weights accordingly. The fused features are obtained by interactively calculating each element in the two text vectors. Specifically, in this example, the fused features are obtained by interactively calculating each element in the first and second text vectors. The specific processing method is as follows:

[0093] Attention scores are calculated using dot product operations. Let V be the first text vector of the image description. d The second text vector of the user's editing command is V. u For the first text vector V d Each element d in j Second text vector V u Each element u in k Calculate attention score j is the first text vector V d The element index in the vector V, k is the second text vector. u The element index in the table, and the attention score, reflect the degree of semantic association between them;

[0094] By normalizing all attention scores, the weight w of each element is obtained. j,k The weight w of each element j,k The calculation formula is:

[0095]

[0096] Among them, score l.m In fact, it refers to the image description vector V. d element d l and V u element u in m The calculated attention score, i.e. In the attention score calculation section, d is used. j The element in the image description vector is used to represent the element in the weight generation part. In the formula, l and m are index variables used to iterate through all possible element pairs. In this example, d is used. l Distinguish specific elements d jThis is because different index variables are needed to represent elements in the image description vector at different computational steps. In the attention score calculation step, j and k are used as indices to calculate specific element pairs (d...). j and u k The attention scores between all possible element pairs are calculated by summing the attention scores during the weight generation step. This requires using more general index variables l and m to iterate through all element pairs, thus calculating the normalized weight w. j,k .

[0097] This weight represents each element d in the first text vector of the image description when generating fused features. j For each element u in the second text vector of the user's editing instructions k The relative importance of elements is determined, and vice versa. Shared attention mechanisms can determine the importance of each element when comprehensively considering the semantics of two texts. These weights reflect the relative importance of the various semantic components in the image description and user editing instructions within the current image editing task.

[0098] In the image latent feature encoding acquisition step, this example focuses on obtaining the image latent features from the input image through an image encoder. This example uses an image variational autoencoder (VAE), which converts the pixel information of the image into an intermediate feature representation, ultimately obtaining a mean vector μ and a variance vector σ. The VAE encodes the image into a latent noise vector in the latent space, representing the key information of the image in a compact form. Compared to the high-dimensional pixel data of the original image, the latent vector typically has a lower dimensionality, which significantly reduces the amount of data required for subsequent processing and improves computational efficiency. This latent representation can be better integrated with subsequent conditional information in terms of both dimensionality and semantics. During image editing, the model can more easily interact with the latent vector using conditional information, thereby adjusting the image features according to editing instructions.

[0099] Taking condition 1 (the first text vector describing the image) as an example, the original dimension of the first text vector is d1. After passing through a fully connected layer with a weight matrix W1, a transformed first vector c1′ = W1c1 is obtained, which is consistent with the dimension d2 of the latent feature vector of the image. Here, c1 is the first text vector. Similarly, a similar linear transformation is performed on the second text vector and the fused features to obtain the second vector c2′ and the third vector c3′. Then, the input image is processed by the encoder of VAE to obtain the latent vector z of the image. This latent vector contains the key feature information of the image and exists in a compact low-dimensional form, which provides a basis for the fusion with the condition information.

[0100] In this example's image editing steps, the key is that during the iterative denoising process of the diffusion model, which processes latent image features and latent noise features, it receives guidance from the multimodal conditional information obtained in the previous steps, thereby generating an image of user-edited quality. The processing steps in this example are as follows:

[0101] First, the conditional information obtained from the first text vector acquisition step, the second text vector acquisition step, and the fusion feature acquisition step is injected into the DiffusionTransformer model via a cross-attention mechanism. The cross-attention mechanism allows the model to dynamically focus on different parts of the conditional information when processing latent image features. For each latent image feature's location (e.g., corresponding to a region or object in the image), the model determines which semantic content in the conditional information should be focused on in the current editing task by calculating its attention weights with each element in the conditional information. For example, if the editing instruction is to change a person's facial expression in an image, the cross-attention mechanism will cause the model to pay high attention to the description of facial expression in the editing instruction, as well as the image description related to facial expression (such as the person's emotional atmosphere), when processing latent features of the person's facial region in the image. This mechanism calculates attention scores and then obtains attention weights through a softmax function.

[0102] As a specific embodiment of the present invention, the specific calculation process in this example is as follows:

[0103] B1: For each element z in the latent vector z b and condition vector c′ a For each element in the set, a = 1, 2, 3, the attention score is... i represents conditional information used to identify different sources, and k is an index variable used to index the condition vector c′. a Elements in;

[0104] B2: By normalizing, we obtain each element z in the latent vector z. b attention weights The calculation formula is:

[0105]

[0106] The purpose of this formula is to perform a weighted summation of the value vectors of the conditional vectors based on the calculated attention weights, thereby obtaining the fused feature representation. The fused image latent features f n(Having the same dimension as the latent vector z) it contains both the original feature information of the image (from the latent vector z) and conditional information (from conditional vectors c1, c2, and c3), thus realizing the relative importance of each element in the latent vector z through the interaction of conditional information and the latent vector. It is obtained by calculating the attention score and normalizing it using the softmax function;

[0107] B3: Apply the attention weights obtained in step B2 Value vector with conditional information Multiplying and summing yields the latent image features f that incorporate conditional information. n Image latent features f n The calculation formula is:

[0108]

[0109] in, Let c′ be the condition vector a The element in c a k is a pair of elements z in the latent vector z b The relative importance parameter, Let c′ be the condition vector a element c a The vector of values ​​for k;

[0110] B4: Denoising is performed on the latent image features (existing in the form of latent noise vectors) and random latent noise (additional noise introduced during the initialization or iteration of the diffusion model) obtained in the latent image feature encoding acquisition step. During the denoising process, the diffusion model receives guidance from conditional information and gradually adjusts the latent noise vector to change in a direction that conforms to the user's editing intention.

[0111] In the forward diffusion process of the diffusion model, the original latent noise vector (the initial latent feature representation obtained from the image encoder) is gradually enriched with noise. This process is represented by the following formula:

[0112]

[0113] Among them, z n,s It is the s-th potential noise vector, where n represents the noise correlation parameter, z n,s-1 It is the potential noise vector from the previous step, a s It is a coefficient that controls the degree of noise addition, ∈ s It is Gaussian noise. The forward diffusion process causes the potential noise vector to gradually lose its original image information, eventually becoming a distribution that is close to pure noise.

[0114] In the reverse denoising process, the goal of the diffusion model is to recover the latent image features that match the user's editing intent from the latent noise vector that is overwhelmed by noise. At this point, the latent noise vector, which incorporates conditional information through the aforementioned cross-attention mechanism, comes into play. The neural network in the diffusion model predicts the noise in each diffusion step and uses this predicted noise to update the estimated value z′ of the latent noise vector. n,s-1 The calculation formula is:

[0115]

[0116] Where, z′ n,s Let be the noise vector at diffusion step s, ∈ θ (z′ n,s (s) represents the latent noise vector z′ predicted by the neural network with parameter θ in the diffusion model at diffusion step s. n,s In this process, conditional information influences noise prediction through a cross-attention mechanism, causing the update of the potential noise vector to proceed in a direction that aligns with the user's editing intent.

[0117] In the reverse denoising process of the diffusion model, conditional information (conditional vectors c1, c2, and c3) is incorporated into the processing of the latent noise vector zn through a cross-attention mechanism. Specifically, in calculating the latent noise vector z... n During the update, the model uses a cross-attention mechanism to consider conditional information. Assuming that in the s-th step of the reverse denoising, the potential noise vector is z... n,s The conditional information vector is c′ a = [c1', c2', c3']. Through the cross-attention mechanism, attention weights based on conditional information are calculated, which in turn affect the prediction of noise.

[0118] Although the condition vector symbol does not appear directly in the formula, the information of the condition vector is incorporated through the cross-attention mechanism during actual calculation (i.e., noise prediction), thereby realizing the influence of condition information on noise prediction.

[0119] The key to the image restoration step is converting the image from the latent space into pixel space to obtain the final output image. In the process of decoding latent features into an image, this example maintains the editing effects achieved through image editing operations in the image editing step. Whether it's modifications to image content (such as adding, deleting, or modifying objects) or adjustments to image style, these changes are accurately reflected in the final generated image. For example, if the color of an object in the image is changed during editing, the object's color will be correctly displayed as the edited color in the output image after image restoration, and other parts of the image will also remain consistent with the editing operation, ensuring the integrity and accuracy of the editing result.

[0120] This invention also provides an image editing system based on multimodal condition adaptation, used in the aforementioned image editing method based on multimodal condition adaptation, comprising:

[0121] First text vector acquisition module: used to acquire the input image, process it to obtain the image description corresponding to the image, process the image description through the editor, and obtain the first text vector based on the image information;

[0122] The second text vector acquisition module is used to acquire the user's editing instructions for the image, process the editing instructions, and obtain a second text vector based on the editing instructions.

[0123] The fusion feature acquisition module is used to assign weights to the first text vector and the second text vector, and obtain fusion features based on the assigned weights and by fusing the first text vector and the second text vector;

[0124] Image latent feature encoding acquisition module: used to compress the input image into the latent space through an image encoder to obtain the image latent features;

[0125] Image editing module: Implemented through a diffusion model, it is used to receive conditional information injected through a cross-attention mechanism. The conditional information includes a first text vector, a second text vector, and fused feature information. At the same time, the diffusion model denoises the latent features and latent noise of the received image. The guidance of the received conditional information gradually generates the image desired by the user during the iterative denoising process.

[0126] Image restoration module: Used to restore the edited image into a pixel-space image using an image decoder.

[0127] In summary, the present invention has the following innovative features:

[0128] 1. This invention can accurately realize editing intentions, whether it is simple attribute modification or complex scene transformation, it can meet the user's editing needs;

[0129] 2. This invention can generate high-quality edited images. Through the architecture of this invention, the image maintains structural and semantic coherence, and the edited elements blend naturally with the original scene without any abrupt or unreasonable combinations.

[0130] 3. This invention enables powerful editing flexibility and controllability. Through cross-attention and shared attention mechanisms, user editing commands can precisely control the image editing process and results. It can accurately operate on every detail of the image (such as the modification of specific parts of an object, local lighting adjustment, etc.), greatly improving the flexibility and controllability of editing.

[0131] 4. This invention exhibits excellent generalization and adaptability. The model components trained on large-scale data (such as VLM and Diffusion Transformer) possess powerful generalization capabilities, enabling them to adapt to image editing tasks of different types, styles, and content. Whether it's natural scenery, portraits, architectural scenes, or abstract artworks, it can effectively handle them. For different editing needs and scene variations (such as different user instructions, differences in image quality, etc.), the model can flexibly adjust its editing strategy, providing stable and high-quality editing results, demonstrating excellent adaptability.

[0132] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. An image editing method based on multimodal condition adaptation, characterized in that, Includes the following steps: The first text vector acquisition steps are as follows: acquire the input image, process it to obtain the image description corresponding to the image, process the image description through an editor, and obtain the first text vector based on the image information; The second text vector acquisition step is to obtain the user's editing instructions for the image, process the editing instructions, and obtain the second text vector based on the editing instructions. The fusion feature acquisition step is as follows: weights are assigned to the first text vector and the second text vector to obtain fusion features based on the assigned weights and by fusing the first text vector and the second text vector; Image latent feature encoding acquisition steps: The input image is compressed into the latent space by an image encoder to obtain the image latent features; Image editing steps: The diffusion model receives conditional information injected through a cross-attention mechanism, which includes a first text vector, a second text vector, and fused features; at the same time, the diffusion model denoises the received latent features and random latent noise of the image, and gradually generates the image desired by the user during the iterative denoising process guided by the conditional information; Image restoration steps: The edited image is restored to a pixel-space image using an image decoder.

2. The image editing method based on multimodal condition adaptation according to claim 1, characterized in that: In the first text vector acquisition step, the image is converted into an image description using a Visual Language Model (VLM). The VLM model, through pre-training, establishes a mapping relationship between image content and natural language description. The training method for the VLM model is as follows: A1: The image encoder and text encoder of the VLM model extract features from the image and text, respectively; A2: The image encoder converts an image into a visual feature vector v, and the text encoder converts text into a text feature vector t; A3: Calculate the similarity score s between positive sample pairs. pos =v T t pos And the similarity score S between negative sample pairs neg =v T t neg , where t pos t is the feature vector of the positive sample text. neg It is the feature vector of the negative sample text, v T This represents the transpose of vector v; A4: Calculate the loss L by comparing the learning loss function. contrastive The calculation formula is: Where N is the number of negative samples, and i is the index variable for summation; A5: Randomly mask some words in the text, and let the model predict the masked words based on the image content and the unmasked words, so that the model can learn the language structure and semantic relationships of the text. A6: Use a text encoder to extract features from the masked text to obtain the masked feature vector t. masked ; A7: Transfer the eigenvector t masked Together with the corresponding image feature vector v, it is input into the decoder or prediction layer of the VLM model to predict the probability distribution p of the masked word. masked ; A8: Calculate the loss of the masked language model using the cross-entropy loss function. For each masked word, the loss is... Among them, y k is the true label of the masked word, and M is the number of masked words. This loss function enables the model to accurately predict the probability distribution of the masked words based on the image content and context information. By optimizing the masked language model loss, the VLM model can generate reasonable language descriptions based on the image content.

3. The image editing method based on multimodal condition adaptation according to claim 2, characterized in that: After acquiring the input image, the VLM model analyzes the image and generates a natural language description to summarize the key elements in the image, including scenes, people and / or objects.

4. The image editing method based on multimodal condition adaptation according to claim 1, characterized in that: In the feature fusion acquisition step, a shared attention mechanism is adopted. By interactively calculating each element in the first and second text vectors, the fusion features are obtained. The specific processing method is as follows: The attention score is calculated using dot product operations, and the first text vector of the image description is V. d The second text vector of the user's editing command is V. u For the first text vector V d Each element d in j Second text vector V u Each element u in k Calculate attention score Attention score reflects the first text vector V d Each element d in j Second text vector V u Each element u in k The degree of semantic connection between them; By normalizing all attention scores, the weight w of each element is obtained. j,k This weight represents the weight of each element d in the first text vector of the image description when generating fused features. j For each element u in the second text vector of the user's editing instructions k The relative importance of.

5. The image editing method based on multimodal condition adaptation according to claim 1, characterized in that: In the image latent feature encoding acquisition step, an image variational autoencoder (VAE) is used to convert the pixel information of the image into an intermediate feature representation. The final image latent features include a mean vector μ and a variance vector σ. The VAE encodes the image into the latent space to obtain a latent noise vector, which represents the key information of the image in a compact form.

6. The image editing method based on multimodal condition adaptation according to claim 5, characterized in that: The original dimension of the first text vector is d1. After passing through a fully connected layer with a weight matrix W1, a transformed first vector c'1 with the same dimension d2 as the latent feature vector of the image is obtained. The first vector c'1 is the product of the first text vector and the weight matrix. Similarly, a similar linear transformation is performed on the second text vector and the fused features to obtain the second vector c'2 and the third vector c'3. Then, the input image is processed by the encoder of VAE to obtain the latent noise vector z of the image.

7. The image editing method based on multimodal condition adaptation according to claim 6, characterized in that: The specific processing methods for image editing steps are as follows: B1: For each element z in the potential noise vector z b and condition vector c' a For each element in the set, a = 1, 2, 3, the attention score is... i represents the identifier of the condition information from different sources, and k is the index variable used to index the condition vector c'. a Elements in; B2: By normalizing, we obtain each element z in the potential noise vector z. b Attention weights and condition vector c' a Attention weights of each element in the vector z for each element in the latent noise vector. B3: Apply the attention weights obtained in step B2 Value vector with conditional information Multiplying and summing yields the latent image features f that incorporate conditional information. n Image latent features f n The calculation formula is: Among them, attention weight Represents the condition vector c' a Each element in the latent noise vector z corresponds to an element z in the latent noise vector z. b The relative importance of c a k is a pair of elements z in the latent vector z b The relative importance parameter, For the condition vector c' a element c a The vector of values ​​for k; B4: Denoising is performed on the latent image features and random latent noise obtained in the image latent feature encoding step. During the denoising process, the diffusion model receives guidance from conditional information and gradually adjusts the latent noise vector to change in a direction that conforms to the user's editing intention. In the forward diffusion process of the diffusion model, the initial latent feature representation of the image obtained from the image encoder is gradually supplemented with random latent noise. This process is expressed by the following formula: Among them, z n,s It is the s-th potential noise vector, where n represents the noise correlation parameter, z n,s-1 It is the potential noise vector from the previous step, a s It is a coefficient that controls the degree of noise addition, ∈ s It is Gaussian noise. The forward diffusion process causes the potential noise vector to gradually lose its original image information, eventually becoming a distribution that is close to pure noise. In the reverse denoising process, the goal of the diffusion model is to recover the latent image features that match the user's editing intent from the latent noise vector that is overwhelmed by noise. The neural network in the diffusion model predicts the noise in each diffusion step and uses the predicted noise to update the estimated value z' of the latent noise vector. n,s-1 The calculation formula is: Among them, z' n,s Let be the s-th potential noise vector before denoising, ∈ θ (z' n,s When denoising the s-th potential noise vector, z' is the corresponding potential noise vector predicted by the neural network with parameter θ in the diffusion model. n,s In this process, conditional information influences noise prediction through a cross-attention mechanism, causing the update of the potential noise vector to proceed in a direction that aligns with the user's editing intent.

8. A multimodal condition adaptation-based image editing system, used to implement the multimodal condition adaptation-based image editing method according to any one of claims 1-7, characterized in that, include: The first text vector acquisition module is used to acquire the input image, process it to obtain the image description corresponding to the image, process the image description through the editor, and obtain the first text vector based on the image information. The second text vector acquisition module is used to acquire the user's editing instructions for the image, process the editing instructions, and obtain a second text vector based on the editing instructions. The fusion feature acquisition module is used to assign weights to the first text vector and the second text vector, and obtain fusion features based on the assigned weights and by fusing the first text vector and the second text vector; Image latent feature encoding and acquisition module: used to compress the input image into the latent space through the image encoder to obtain the image latent features; Image editing module: Implemented through a diffusion model, it receives conditional information injected via a cross-attention mechanism. The conditional information includes a first text vector, a second text vector, and fused features. Simultaneously, the diffusion model denoises the received image's latent features and random latent noise, and gradually generates the user's desired image during the iterative denoising process guided by the conditional information. Image restoration module: Used to restore the edited image into a pixel-space image using an image decoder.

Images